1195893 - Change aws instance type for Android 4.3 Opt mochitests (M1..M16) to c3.xlarge

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1189784, I am trying to enable dom/media mochitests on Android 4.3. Those tests have always been skipped on Android 4.3, since most of those tests fail intermittently on that platform. Many of the test failures are timeouts; I have tried increasing the timeout thresholds, but this has not helped much -- it seems tests continue to hang. I suspect there is not enough cpu available to run the emulator + test + media support.

However, if I change the aws instance type to c3.xlarge, then most of the dom/media tests pass reliably.

Can we run Android 4.3 Opt mochitests (M1 .. M16) on c3.xlarge?

There are other limitations preventing us from running dom/media mochitests on Android 2.3 so I am not suggesting any changes for 2.3.

Android 4.3 Debug mochitests already run on c3.xlarge so I am not suggesting any changes for 4.3 Debug.

(fwiw, I am starting to think all emulator tests should run on c3.xlarge, since we seem to keep running into problems like this. The m1.medium seems to be just barely capable of running the emulator -- anything extra and we run into trouble. I don't have a specific need beyond dom/media on 4.3 Opt though.)

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1195893cloud.patch

Comment 2

[Kim Moir [:kmoir] ET]

Attachment: bug1195893.patch

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: bug1195893cloud.patch

Comment 4

[Kim Moir [:kmoir] ET]

Comment on attachment 8650083 [details] [diff] [review]
bug1195893.patch

the builder diff is just the same jobs but in a different order.  No actual job names are being changed, just the underlying instance type

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 8650083 [details] [diff] [review]
bug1195893.patch

Review of attachment 8650083 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/mobile_config.py
@@ +2391,5 @@
>  for suite in ANDROID_4_3_MOZHARNESS_DICT: 
>      if suite[0].startswith('plain-reftest'):
>          ANDROID_4_3_C3_DICT['opt_unittest_suites'].append(suite)
>          ANDROID_4_3_C3_DICT['debug_unittest_suites'].append(suite)
> +    elif re.match(r"^mochitest-\d", suite[0]):

I'm hating our configs more and more when I see stuff like this (regex's based on suite name, is scary)

@@ +2395,5 @@
> +    elif re.match(r"^mochitest-\d", suite[0]):
> +        ANDROID_4_3_C3_DICT['opt_unittest_suites'].append(suite)
> +        ANDROID_4_3_C3_DICT['debug_unittest_suites'].append(suite)
> +        ANDROID_4_3_C3_TRUNK_DICT['debug_unittest_suites'].append(suite)
> +    elif suite[0].startswith('mochitest') and not (re.match(r"^mochitest-\d", suite[0])):

can save some minor computational time and omit |and not...| since we'll never get here if it did match (due to the condition your adding above)

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1195893-2.patch

patch with fixes from review

Comment 7

[Kim Moir [:kmoir] ET]

verified that it's working on treeherder

will keep an eye on emulator64 pool and see if the size needs to be increased

Comment 8

[Kim Moir [:kmoir] ET]

Note to future self
run
for f in `find . -name '*.json'` ./configs/watch_pending.cfg; do python -m json.tool $f; done
on the watch_pending.cfg before submitting patch

because this caused watch pending to fail and new spot instances not to be created

Fix
https://github.com/mozilla/build-cloud-tools/commit/5f23650624fc4a34b051d8b045f7363e3b728355