1164866 - Android arm emulator tests should log results as they happen

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Geoff Brown [:gbrown]]

In the initial work on Android x86 emulators, we decided to run up to 4 test jobs (in up to 4 emulators) simultaneously on each computer, to fully utilize the hardware available.

The Android arm emulator tests - 2.3 and 4.3 - use the same android_emulator_unittest.py mozharness script and theoretically support the same capability, but in practice only ever run one test job (in one emulator) at a time.

Because android_emulator_unittest.py must allow for parallel test runs, the output from the test harness is buffered in a file so that test results are not intermingled in the test log. Each file is parsed and dumped only once tests are complete.

As a result, in bugs like bug 1158206 where there is some sort of hang, it is unclear which tests have run, because all the output has been buffered.

Also, the mozharness log time stamps are pretty much the same for the entirety of the test run (since all the buffered output is logged at once, at the end of the job).

I don't see how these limitations can be overcome for x86, but we can do better for arm.

Comment 1

[Geoff Brown [:gbrown]]

Strategy:
 - copy android_emulator_unittest.py to androidx86_emulator_unittest.py
 - update buildbot config to use androidx86_emulator_unittest.py for x86 jobs
 - simplify android_emulator_unittest.py to support one job at a time
 - parse test output as it happens

Comment 2

[Geoff Brown [:gbrown]]

Attachment: copy existing script to x86 version

Simple copy of android_emulator_unittest.py to androidx86_emulator_unittest.py.

Comment 3

[Geoff Brown [:gbrown]]

Attachment: update buildbot config to use new x86 script for x86 jobs only

Comment 4

[Geoff Brown [:gbrown]]

Attachment: simplify android_emulator_unittest.py

This simplifies android_emulator_unittest.py, stripping out the lists and loops used to support multiple emulators and multiple test suites. Test harness output is still buffered.

Comment 5

[Geoff Brown [:gbrown]]

Attachment: do not buffer test harness output

This updates the way we run the test harness and report output. This is now similar to the way "normal" mozharness scripts work -- output is passed to the log parser as it happens.

Comment 6

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0eeb9fffb720

Notice the log timestamps...

http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/[e-mail]-0eeb9fffb720/try-android-api-9/try_ubuntu64_vm_mobile_test-xpcshell-1-bm67-tests1-linux64-build56.txt.gz

09:57:39     INFO -  TEST-PASS | chrome/test/unit/test_abi.js | took 4465ms
09:57:39     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
09:57:40     INFO -  TEST-START | chrome/test/unit/test_bug292789.js
09:57:44     INFO -  TEST-PASS | chrome/test/unit/test_bug292789.js | took 3722ms
09:57:44     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
09:57:45     INFO -  TEST-START | chrome/test/unit/test_bug380398.js
09:57:49     INFO -  TEST-PASS | chrome/test/unit/test_bug380398.js | took 3762ms

Instead of the normal...

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-9/1431541543/mozilla-central_ubuntu64_vm_mobile_test-xpcshell-1-bm54-tests1-linux64-build29.txt.gz

13:33:26     INFO -  TEST-PASS | chrome/test/unit/test_abi.js | took 4346ms
13:33:26     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
13:33:26     INFO -  TEST-START | chrome/test/unit/test_bug292789.js
13:33:26     INFO -  TEST-PASS | chrome/test/unit/test_bug292789.js | took 3562ms
13:33:26     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
13:33:26     INFO -  TEST-START | chrome/test/unit/test_bug380398.js
13:33:26     INFO -  TEST-PASS | chrome/test/unit/test_bug380398.js | took 3631ms

Comment 7

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/mozharness/rev/560859d88ba8

Comment 8

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6272ca3dc104

Comment 9

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/6272ca3dc104

Comment 10

[Release Engineering SlaveAPI Service]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

Comment 11

[Release Engineering SlaveAPI Service]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

Comment 12

[Geoff Brown [:gbrown]]

Before I can land the other patches here, particularly the buildbot config change, I need mozharness bumped to 560859d88ba8 (or later) on aurora, beta, and any other trees where Android x86 tests are run. :ryanvm, can you do this?

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/18286de4be87

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/e71f34c55d27

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/f2ef3e1dadaf

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr38/rev/10f8fc9e6960

Doesn't run Android tests, but might as well keep them in sync if possible.

Comment 17

[Geoff Brown [:gbrown]]

Awesome. Thanks.

Comment 18

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/buildbot-configs/rev/76b11421370e

Comment 19

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/76b11421370e

Comment 20

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/mozharness/rev/f374ed409097
https://hg.mozilla.org/build/mozharness/rev/d097dc489cdf

Comment 21

[Geoff Brown [:gbrown]]

The last changes here started running on m-c 2015-05-22.