1243408 - Autophone - update test manifests to better utilize new devices and improve test coverage.

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

With the addition of new devices and the improvements in job completion times in Autophone, it is time to better allocate our devices across tests and branches.

Comment 1

[Bob Clary [:bc] (inactive)]

with 2min/tpn, 2min/tsvg, 2?min/install we are looking at 6min/build and a max of 240 builds per day. We can handle that on one device I think. A recent bread down of builds I did was:

2016-01-26

fx-team          16
mozilla-aurora    2
mozilla-beta      4
mozilla-central   3
mozilla-inbound 101
b2g-inbound       2
                ---
                128

jmaher, I'll do a quick patch to turn on talos for mozilla-inbound and fx-team for nexus-6p-1.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: bug-1243408-v1.patch

Comment 3

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/c534f066abb138c0fa92707cbad5847cdb973b27

leave this open for additional changes.

Comment 4

[Bob Clary [:bc] (inactive)]

totally forgot tpn.
https://github.com/mozilla/autophone/commit/7490d428562405a522c00ca743d31e9da708646b

Comment 5

[Bob Clary [:bc] (inactive)]

adding nexus-6p-2 to the talos testing and cleaning up the manifests related to nexus-6p devices.

https://github.com/mozilla/autophone/commit/d2e371f7ebb5dcb7624dd5bc3967435fdec15900

Comment 6

[Bob Clary [:bc] (inactive)]

I am going to temporarily shut autophone down to run a simple test with s1s2 remote tests on all devices to get a picture of their network connectivity before assigning tests to specific devices. ETA should be well less than an hour.

Comment 7

[Bob Clary [:bc] (inactive)]

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=ec30bdea196a&filter-tier=1&filter-tier=2&filter-tier=3&exclusion_profile=false&filter-searchStr=autophone

lacking nexus-6p-2 and nexus-s-10 due to device disconnections. Overall, looks better than I thought it would. The nexus-6p-1 device did not appear to have any issues with the remote tests. Perhaps there is some other reason for the apparent disconnections.

Either way, it appears there is no restriction on assigning devices to remote s1s2 tests as far as this limited test can tell.

The servers are back to their normal test manifests.

Comment 8

[Bob Clary [:bc] (inactive)]

Attachment: dump-manifests.py

ad hoc script I used to compute the test coverage and device loads. Not meant to be production quality.

Comment 9

[Bob Clary [:bc] (inactive)]

Attachment: before-coverage.txt

Coverage of existing test manifests.

Comment 10

[Bob Clary [:bc] (inactive)]

Attachment: bug-1243408-production-manifests.patch

Comment 11

[Bob Clary [:bc] (inactive)]

Attachment: after-coverage.txt

The report has 3 sections.

1. repo by test by devices.
   This section is useful in seeing if a repo has sufficient coverage from the
   various classes of device.
2. devices by test by repo.
   This section also shows the estimated time each device has been commited per
   day. This is based on an average number of builds per day per repo except in
   the case of mozilla-inbound where the value is a close to the maximum seen
   since Sept 2015.
3. test by repo by device

While we have many more devices available now than we did with the mac mini, we are still limited by the large number of potential mozilla-inbound builds per day and:

a) we only have 2 nexus 6 and 2 nexus 9 devices.
b) nexus s devices currently take far too long to execute due to the lack of a
   native recursive chmod. I hope to have a solution for this later. Of course,
   killing off Android 2.3 will kill off the need for the nexus s devices.

Note that I have dropped b2g-inbound. I believe this set of manifests achieves
a much better distribution of devices over the possible tests and repos.

When we get the remainder of the nexus 6p devices installed, we will be able
to increase their coverage for non-talos tests.

Comment 12

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/6e5755bf731c5c143e994e1d361e8afb3b7ccb42

PS. Note that I have limited production running of mochitest-dom-media to nexus s due to its lack of failures and time outs. mochitest-dom-media is available via try commit messages like:

try: -b o -p android-api-9,android-api-15 -u autophone-mochitest-dom-media -t none

note it appears -t none is required.

deployed autophone-1 2016-02-12 22:24
         autophone-2 2016-02-12 22:28
         autophone-3 2016-02-12 22:28