1378056 - 8.32% remote-nytimes (android-4-4-armv7-api15) regression on push 0e12b505a2385840a6ae04fd444b813f23b0e698 (Thu Jun 29 2017)

Status: RESOLVED WORKSFORME

(Core :: Audio/Video: Playback, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected an autophone (Android) regression from push:

https://hg.mozilla.org/integration/autoland/pushloghtml?changeset=0e12b505a2385840a6ae04fd444b813f23b0e698

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

  8%  remote-nytimes summary android-4-4-armv7-api15 opt      2,293.25 -> 2,484.03


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=7611

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://wiki.mozilla.org/EngineeringProductivity/Autophone

Comment 1

[Ionuț Goldan [:igoldan]]

:jwwang Could you take a look over this regression and estimate a fix time for it?

Comment 2

[Jean-Yves Avenard [:jya]]

there's no way the reported push caused a regression as indicated.

So I would question the validity of that test first, it's either faulty, invalid noise and temporary (like most of those perf tests) or something else got changed infrastructure wise.

Comment 3

[JW Wang [:jwwang] [:jw_wang]]

Where is the test case of remote-nytimes?

Comment 4

[Bob Clary [:bc] (inactive)]

The phone nexus-5-3 had issues and was disconnected around 2017-06-22. See bug 1375601. I was on site and changed the servers to run Fedora 25 on 2016-06-28 - 2016-06-29 when I reconnected the phone. See Bug 1377108.

The test files for the nytimes aren't publicly available since they are of copyrighted materials. You can find a copy linked from https://mana.mozilla.org/wiki/display/ateam/Autophone.

Looking at the graph for this phone and test we see the period where there was no testing following by the discontinuous change after the first data point when it was reconnected:

http://phonedash.mozilla.org/#/2017-06-18/2017-06-29/binning=repo-phonetype-phoneid-test_name-cached_label-metric&autoland=on&nexus-5-3=on&remote-nytimes=on&throbberstart=on&throbberstop=on&first=on&second=on&rejected=norejected&errorbars=errorbars&errorbartype=standarderror&valuetype=median

My first thought is that this is not related to the changeset, but is more related to the phone's state after being rested. Expanding the date range a bit you see something that looks like a real change which was backed out then landed again and backed out again:

http://phonedash.mozilla.org/#/2017-06-18/2017-07-04/binning=repo-phonetype-phoneid-test_name-cached_label-metric&rejected=norejected&errorbars=errorbars&errorbartype=standarderror&valuetype=median&remote-nytimes=on&throbberstart=on&throbberstop=on&first=on&second=on&autoland=on&nexus-5=on&nexus-5-1=on&nexus-5-3=on&nexus-5-5=on&nexus-5-6=on

Looking at these changes however does not show a consistent pattern of changesets landing and being backed out. In fact, there does appear to be a correlation across the beta and release branches to the changes on the trunk branches which would only be the case if this is a device issue unrelated to the changesets being tested. I will trigger the missing tests for this device from 2017-06-22 to 2017-06-29 and we will see how testing those earlier changesets corresponds to the current test results.

Comment 5

[Bob Clary [:bc] (inactive)]

on autophone-2:
python trigger_runs.py --build-location=tinderbox --repo=mozilla-central --buildtype=opt --test=autophone-s1s2 --device=nexus-5-3 --first-revision=2403cb851fe3e56b9018eaa645c78e913d927812 --last-revision=9af23c413a1f8d337b19b4f8450e241e91b71136

queued up 27 jobs

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&fromchange=2403cb851fe3e56b9018eaa645c78e913d927812&tochange=9af23c413a1f8d337b19b4f8450e241e91b71136

20170622160919 - 20170629012006
http://phonedash.mozilla.org/#/2017-06-21/2017-07-04/binning=repo-phonetype-phoneid-test_name-cached_label-metric&rejected=norejected&errorbars=errorbars&errorbartype=standarderror&valuetype=median&remote-nytimes=on&throbberstart=on&throbberstop=on&first=on&second=on&mozilla-central=on&nexus-5=on&nexus-5-1=on&nexus-5-3=on&nexus-5-5=on

Comment 6

[Bob Clary [:bc] (inactive)]

From what I can tell this change was transient. There appear to be two periods where autoland/inbound both show an increase across the board at the same times. A real regression would have appeared on one first then the other when the merge occured. nexus-5-3 reverted back to the previous base line after July 1 though it appears to now have bi-modal behavior. nexus-5-3 may be ending its end of life.

I'll mark this wfm and we can move on.