1416789 - Autophone - Disable performance testing on Nexus 6P?

Status: RESOLVED WONTFIX

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: nexus-6p-assignments.txt

The Nexus 6p devices running performance tests have had issues recently either due to the excessive load placed on them or just due to device age. Most of the devices are approaching 2 years in age.

igoldan has been dealing with false alerts due to device specific regressions. Some of the 6p devices have had spurious results which I have manually excluded but which are very apparent when comparing graphs.

[with excluded data]
<http://phonedash.mozilla.org/#/2017-11-06/2017-11-13/binning=repo-phonetype-phoneid&rejected=rejected&errorbars=noerrorbars&errorbartype=standarderror&valuetype=median&geckoview-e10s-nytimes=on&geckoview-nytimes=on&remote-blank=on&remote-nytimes=on&remote-twitter=on&throbberstart=on&throbberstop=on&throbbertime=on&first=on&second=on&autoland=on&mozilla-beta=on&mozilla-central=on&mozilla-inbound=on&mozilla-release=on&nexus-4=on&nexus-4-09=on&nexus-5=on&nexus-5-02=on&nexus-5-03=on&nexus-5-04=on&nexus-5-05=on&nexus-6p=on&nexus-6p-08=on&nexus-6p-10=on&nexus-6p-11=on&nexus-6p-12=on&nexus-6p-13=on&pixel=on&pixel-03=on&pixel-04=on&pixel-11=on>

[without excluded data]
<http://phonedash.mozilla.org/#/2017-11-06/2017-11-13/binning=repo-phonetype-phoneid&rejected=norejected&errorbars=noerrorbars&errorbartype=standarderror&valuetype=median&geckoview-e10s-nytimes=on&geckoview-nytimes=on&remote-blank=on&remote-nytimes=on&remote-twitter=on&throbberstart=on&throbberstop=on&throbbertime=on&first=on&second=on&autoland=on&mozilla-beta=on&mozilla-central=on&mozilla-inbound=on&mozilla-release=on&nexus-4=on&nexus-4-09=on&nexus-5=on&nexus-5-02=on&nexus-5-03=on&nexus-5-04=on&nexus-5-05=on&nexus-6p=on&nexus-6p-08=on&nexus-6p-10=on&nexus-6p-11=on&nexus-6p-12=on&nexus-6p-13=on&pixel=on&pixel-03=on&pixel-04=on&pixel-11=on>

nexus-6p-12 and nexus-6p-13 are typically overloaded and fall significantly behind. They are also some of the most common devices to produce extremely out of range results.

I have made a request for 10 additional 6p devices in Bug 1410514.

We have several choices which are not mutually exclusive

* Disable Nexus 6p performance testing now.

* Reduce the load by removing autoland and mozilla-inbound testing.

* Reduce the load by removing some but not all tests.

* Send the devices to me to see if the devices can be rehabbed by reflashing Android etc.

I've attached a list of test assignments for nexus 6p devices.

igoldan: It is ok with me to disable the devices. Just let us know if they are too much trouble or if they do not provide useful results.

snorp: Reducing tests would help. If you can tell us which tests you don't care about we can disable them immediately.

Comment 1

[Bob Clary [:bc] (inactive)]

Autophone is going away. Resolving these to wontfix.