1522221 - Android 4.3 tests take a long time to run

Status: RESOLVED FIXED

(Testing :: General, enhancement, P3)

Description

[Geoff Brown [:gbrown]]

When comparing total test task durations per test platform, Android 4.3 stands out at the top, or near the top of any list. These are the "Android 4.3 API16+ opt" and "Android 4.3 API16+ debug" tests seen on treeherder, running tests against Firefox for Android.

These tests run on an Android armv7 emulator, on aws. Code run in the emulator is slow. Most tasks run with a tc worker type of gecko-t-linux-xlarge: an m3.xlarge or c3.xlarge aws instance type, more expensive than many desktop test tasks.

Some test suites are particularly slow. On debug, the jsreftest suite runs in 100 chunks and still most chunks require over 60 minutes to run: over 100 hours duration per jsreftest run.

Significant effort has been put into trying to find a more efficient version of the arm emulator, but without success. Newer versions of the Android emulator and newer versions of Android seem to run even slower than what is in use.

An alternative is to run on the Android x86 emulator, with kvm acceleration. We do this for the "Android 7.0 x86 opt" tests, which run on packet.net. Currently jsreftest runs there in a little over 2 hours. But that test platform was put together for geckoview testing and those tests run geckoview test apps. So the existing tests are not a replacement for Android 4.3: Android 4.3 is testing Firefox for Android; Android 7.0 x86 is testing geckoview.

Could we test Firefox for Android on the Android 7.0 x86 platform? Probably, but there would be considerable effort required to "green" tests on the new platform and that's probably not justified if Firefox for Android is near end of life. Our current strategy is to wait for Firefox for Android to move to ESR.

Comment 1

[Geoff Brown [:gbrown]]

In the mean time, it may be possible to turn off a few tests, but only once significantly similar tests are proven on Android 7.0 x86: see bug 1473368. For example, Android 4.3 geckoview-junit should be redundant once Android 7.0 x86 geckoview-junit is running green.

Comment 2

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #0)

When comparing total test task durations per test platform, Android 4.3 stands out at the top, or near the top of any list.

I looked for evidence of this and was a little surprised by the actual numbers. I used a custom script to collect treeherder data for the most recent 300 pushes on each tree. Total task durations for Android are similar to those for Windows and less than Linux. Here is the breakdown by os for mozilla-central:

linux tests
GROUP TOTAL: 110643 matching jobs, 164833895 seconds (45787 hours)

windows tests
GROUP TOTAL: 80684 matching jobs, 106601627 seconds (29611 hours)

osx tests
GROUP TOTAL: 18586 matching jobs, 24488592 seconds (6802 hours)

android arm tests
GROUP TOTAL: 32616 matching jobs, 83426847 seconds (23174 hours)

This makes sense because Android 4.3 only has opt/debug variants, while linux has linux32/opt and debug, linux64/opt/debug/pgo/asan, linux64-qr/opt and debug. Also, Android 4.3 does not run all test suites (does not run wpt or mochitest-browser-chrome notably).

Comment 3

[Geoff Brown [:gbrown]]

Fennec moving to esr68 discussion: https://groups.google.com/forum/#!topic/mozilla.dev.platform/k-irJtmCcqg

Note "Fennec Code Cleanup & Refactoring" in the referenced doc: https://docs.google.com/document/d/1oRPkwP3l7QzdQYj0Wn7d_3EfTZaakocA_i_pGKlG0dI/edit?usp=sharing.

While not all details are final, we should be running far fewer Android 4.3 tests in the near future.

Comment 4

[Geoff Brown [:gbrown]]

Android 4.3 tests no longer run on mozilla-central.