1438665 - Experiment with running android mochi/reftests on Talos moonshot hardware

Status: RESOLVED WORKSFORME

(Testing :: General, defect, P1)

Description

[Geoff Brown [:gbrown]]

A couple of weeks ago I tried running Android tests on an x86 emulator running on the Talos moonshot hardware. Let's record those results here.

Comment 1

[Geoff Brown [:gbrown]]

I used a new-ish emulator and Android 7.0 x86 avd for this (same as in bug 1438351, on packet.net):

[
  {
    "size": 445624230,
    "visibility": "public",
    "digest": "fb951185e21ba6124d8c8ba42202613312401c65e56d52480cc08c87816936b9dd00024f4fc00183ee0e199c91ed5b9209867983c1d13daf5b34bcdc14ee9fb6",
    "algorithm": "sha512",
    "filename": "AVDs-x86-android-7.0-build-2017-12-08.tar.gz",
    "unpack": true
  }
]

        [
        {
        "size": 237849484,
        "visibility": "internal",
        "digest": "596db3063758aea49f441ed3dccc5429408e037ba953ab8ea676969c12ed267416a7b1f7d61ad5a06fd2c0fe234131843bd9c57a33b8dd8677aa0b25bbb799d7",
        "algorithm": "sha512",
        "filename": "android-sdk_r26b-linux.tar.gz",
        "unpack": "True"
        }
        ]

and ran with taskcluster and the regular mozharness support, but notably outside of docker.

Comment 2

[Geoff Brown [:gbrown]]

Try run from Feb 5: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9eb6c775566f9cd69f792c0de207500e2f9802c5

This is a little stale compared to the packet.net results, so I'll reproduce with a current build (and changes like run-by-manifest).

Comment 3

[Geoff Brown [:gbrown]]

I had almost no trouble running with kvm acceleration (just needed a permission group change to access /dev/kvm) and found the emulator can run with -gpu host.

Comment 4

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4b0429f1289a0e8ac4688c085270c043a152745a

Notice:
 - there are occasional failures to start the emulator (similar to current arm emulator)
 - crashtests run clean and very fast: 8 minutes
 - jsreftests run clean and fast: about 100 minutes (across 4 chunks)
 - plain reftests run with a few failures in about 50 minutes (across 2 chunks)
 - plain mochitests and mochitest-chrome have some failures and crashes and often fail to complete because the emulator becomes unresponsive
 - mochitests that do run complete fairly fast. for example, dom/tests/mochitest/dom-level2-html completed in 1:46 (compare to about 3:10 typical for arm)
 - robocop tests typically fail with a security exception
 - xpcshell tests typically fail pushing to the device; if that succeeds, tests often crash

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #4)
>  - mochitests that do run complete fairly fast. for example,
> dom/tests/mochitest/dom-level2-html completed in 1:46 (compare to about 3:10
> typical for arm)

Sorry, "about 3:10" for arm is completely wrong. It's much longer on arm, so:

 - mochitests that do run complete very fast. for example, dom/tests/mochitest/dom-level2-html completed in 1:46 (compare to about 50:10 typical for arm -- about 25x faster)

Comment 6

[Geoff Brown [:gbrown]]

This worked out great: Some of the fastest android test results I've ever seen -- comparable to desktop -- but this solution doesn't scale well, so while these experiments were informative, this is a dead-end in practical, production terms.

At any rate, I don't have plans for further experiments on moonshot hardware.