935214 - Consolidate Android x86 test chunks and sets once kvm supported

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Once kvm support is available for Android x86 emulator tests, run times for mochitests and reftests (including crashtests and jsreftests) will be reduced dramatically.

Currently, we run several test suites as multiple chunks to avoid 60 minute timeouts. Post-kvm, we can consolidate many chunks without risking timeouts.

With fewer chunks, we can consolidate the sets too.

With kvm:
 - all mochitests can be run in one chunk, but we sometimes run out of memory, so better to run as 2 chunks;
 - it is still probably better to run mochitest-gl separately, since it is sometimes problematic and we run mochitest-gl separately on Android 4.0;
 - all jsreftests can be run in one chunk;
 - all reftests can be run in one chunk, but we sometimes run out of memory, so better to run as 2 or 3 chunks;
 - all crashtests can be run in one chunk;
 - robocop requires 2 chunks;
 - all xpcshell tests can be run in one chunk (still).

So we need at least 10 tests:
  M1 M2 M-gl JSR R1 R2 C RC1 RC2 X

We may as well split that into 12 tests to give 3 full sets of 4. I suggest:

  S1 { M1 M2 M-gl JSR }    S2 { R1 R2 R3 C }     S3 { RC1 RC2 RC3 X }

Comment 1

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: (1) consolidate test chunks

I am not very familiar with this code. If I got it all wrong, I'm happy to scrap it -- just trying to be helpful!

Comment 2

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: (2) consolidate test sets

Comment 3

[Geoff Brown [:gbrown] (pto Apr 11-18)]

I put the first set in ANDROID_X86_MOZHARNESS_DICT and the others in ANDROID_X86_NOT_GREEN_DICT arbitrarily. I expect almost all tests are not green today but should be greenable in a couple of days once kvm and the new emulator are deployed.

Comment 4

[Armen [:armenzg]]

Comment on attachment 827629 [details] [diff] [review]
(1) consolidate test chunks

Review of attachment 827629 [details] [diff] [review]:
-----------------------------------------------------------------

r=+ pending deploying the kvm support first.

Comment 5

[Armen [:armenzg]]

Comment on attachment 827630 [details] [diff] [review]
(2) consolidate test sets

Review of attachment 827630 [details] [diff] [review]:
-----------------------------------------------------------------

r=+ pending deploying the kvm support first.

Comment 6

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(1) https://hg.mozilla.org/build/mozharness/rev/527c757f5db9
(2) https://hg.mozilla.org/build/buildbot-configs/rev/e641038ee358

Comment 7

[John Hopkins (:jhopkins)]

in production