1019824 - Move emulator gaia-ui-tests on cedar to faster AWS nodes

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

The emulator gaia-ui-tests currently running on cedar run too slowly to be useful; they are all timing out after 5 minutes of activity per test, since the VM they're run on isn't powerful enough to run them faster.

Let's experiment and move the cedar runs to IX hardware and see if they work well in that environment.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule gaia-ui-test on emulator on ubuntu64 hw slaves on cedar,

I'm not sure we've ever run emlator tests on this config, so we'll see what happens.

Comment 2

[Jordan Lund (:jlund)]

Comment on attachment 8435077 [details] [diff] [review]
Schedule gaia-ui-test on emulator on ubuntu64 hw slaves on cedar,

Review of attachment 8435077 [details] [diff] [review]:
-----------------------------------------------------------------

oh boy, TIL it might be possible that mozilla-tests/b2g_config.py is harder to comprehend than mozilla-tests/config.py :)

so I think I'm up to speed. It looks like there is a small typo and I just have some follow up questions (see in-line)

::: mozilla-tests/b2g_config.py
@@ +1021,5 @@
>          'unittest-env': {'DISPLAY': ':0'},
>          'enable_opt_unittests': True,
>          'enable_debug_unittests': True,
> +        'ubuntu64_hw-b2g-emulator': {
> +            'opt_unittest_suites': []],

extra closing square bracket typo :)

@@ +1611,4 @@
>  BRANCHES['cedar']['platforms']['emulator']['ubuntu64_vm-b2g-emulator']['opt_unittest_suites'] = \
>      MOCHITEST + CRASHTEST + XPCSHELL + MARIONETTE + JSREFTEST + GAIA_UI
>  BRANCHES['cedar']['platforms']['emulator']['ubuntu64_vm-b2g-emulator']['debug_unittest_suites'] = MOCHITEST_EMULATOR_DEBUG[:] + REFTEST + CRASHTEST + MARIONETTE + XPCSHELL
> +BRANCHES['cedar']['platforms']['emulator']['ubuntu64_hw-b2g-emulator']['opt_unittest_suites'] = GAIA_UI

so if this is just for GAIA_UI, do we need to add the 'marionette-webapi' in the 'suite_config' above?

also, If these suites are timing out on the vm slaves, do we want to do disable them or are you wanting to run them on both for testing?

Comment 3

[Jonathan Griffin (:jgriffin)]

Attachment: Schedule gaia-ui-test on emulator on ubuntu64 hw slaves on cedar,

Addressed review comments; removed marionette-webapi from suite_config.  Since this is only running on cedar, I'd like to them side-by-side very temporarily so I can compare results, and assuming they run better on IX slaves, I'll turn off the vm runs.

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8435814 [details] [diff] [review]
Schedule gaia-ui-test on emulator on ubuntu64 hw slaves on cedar,

Review of attachment 8435814 [details] [diff] [review]:
-----------------------------------------------------------------

sgtm! :)

Comment 5

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8435814 [details] [diff] [review]
Schedule gaia-ui-test on emulator on ubuntu64 hw slaves on cedar,

https://hg.mozilla.org/build/buildbot-configs/rev/38b85099ad1e

Comment 6

[Simone Bruno [:simone]]

Live with reconfig on 2014-06-11 08:47 PT

Comment 7

[Jonathan Griffin (:jgriffin)]

The runs on cedar are failing because they're being launched without a mozharness config:

========= Started '/tools/buildbot/bin/python scripts/scripts/marionette.py ...' failed (results: 2, elapsed: 0 secs) (at 2014-06-11 23:35:05.750262) =========
/tools/buildbot/bin/python scripts/scripts/marionette.py --blob-upload-branch cedar --download-symbols ondemand
 in dir /builds/slave/talos-slave/test/. (timeout 1800 secs) (maxTime 7200 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', 'scripts/scripts/marionette.py', '--blob-upload-branch', 'cedar', '--download-symbols', 'ondemand']
23:35:05     INFO - MultiFileLogger online at 20140611 23:35:05 in /builds/slave/talos-slave/test
23:35:05    FATAL - You need to specify a --marionette-address for non-emulator tests! (Try --marionette-address localhost:2828 )
23:35:05    FATAL - Running post_fatal callback...
23:35:05    FATAL - Exiting -1

https://tbpl.mozilla.org/php/getParsedLog.php?id=41582225&tree=Cedar&full=1

Jordan, any idea what's wrong here?

Comment 8

[Jordan Lund (:jlund)]

(In reply to Jonathan Griffin (:jgriffin) from comment #7)
> The runs on cedar are failing because they're being launched without a
> mozharness config:
> 
> ========= Started '/tools/buildbot/bin/python scripts/scripts/marionette.py
> ...' failed (results: 2, elapsed: 0 secs) (at 2014-06-11 23:35:05.750262)
> =========
> /tools/buildbot/bin/python scripts/scripts/marionette.py
> --blob-upload-branch cedar --download-symbols ondemand
>  in dir /builds/slave/talos-slave/test/. (timeout 1800 secs) (maxTime 7200
> secs)
>  watching logfiles {}
>  argv: ['/tools/buildbot/bin/python', 'scripts/scripts/marionette.py',
> '--blob-upload-branch', 'cedar', '--download-symbols', 'ondemand']
> 23:35:05     INFO - MultiFileLogger online at 20140611 23:35:05 in
> /builds/slave/talos-slave/test
> 23:35:05    FATAL - You need to specify a --marionette-address for
> non-emulator tests! (Try --marionette-address localhost:2828 )
> 23:35:05    FATAL - Running post_fatal callback...
> 23:35:05    FATAL - Exiting -1
> 
> https://tbpl.mozilla.org/php/getParsedLog.php?id=41582225&tree=Cedar&full=1
> 
> Jordan, any idea what's wrong here?

oops. we missed a typo in your patch:
    http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/b2g_config.py#1053
    s/suite_confg/suite_config/

Comment 9

[Jordan Lund (:jlund)]

Attachment: 140612_Bug_1019824_gaia_ui_hw-confg-typo-fix.patch

Comment 10

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8439321 [details] [diff] [review]
140612_Bug_1019824_gaia_ui_hw-confg-typo-fix.patch

Review of attachment 8439321 [details] [diff] [review]:
-----------------------------------------------------------------

Doh!  Thanks for spotting that.  I completely missed that typo even though I was staring at that code trying to figure out what was wrong.

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8439321 [details] [diff] [review]
140612_Bug_1019824_gaia_ui_hw-confg-typo-fix.patch

np!

https://hg.mozilla.org/build/buildbot-configs/rev/2ee0d5c205fb

Comment 12

[Jordan Lund (:jlund)]

buildbot-config patch live in production :)

Comment 13

[Rail Aliiev [:rail]]

TC should solve this.