Closed Bug 1164866 Opened 9 years ago Closed 9 years ago

Android arm emulator tests should log results as they happen

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

defect
Not set
normal

Tracking

(firefox38.0.5 fixed, firefox39 fixed, firefox40 fixed, firefox-esr38 fixed)

RESOLVED FIXED
Tracking Status
firefox38.0.5 --- fixed
firefox39 --- fixed
firefox40 --- fixed
firefox-esr38 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(4 files)

In the initial work on Android x86 emulators, we decided to run up to 4 test jobs (in up to 4 emulators) simultaneously on each computer, to fully utilize the hardware available.

The Android arm emulator tests - 2.3 and 4.3 - use the same android_emulator_unittest.py mozharness script and theoretically support the same capability, but in practice only ever run one test job (in one emulator) at a time.

Because android_emulator_unittest.py must allow for parallel test runs, the output from the test harness is buffered in a file so that test results are not intermingled in the test log. Each file is parsed and dumped only once tests are complete.

As a result, in bugs like bug 1158206 where there is some sort of hang, it is unclear which tests have run, because all the output has been buffered.

Also, the mozharness log time stamps are pretty much the same for the entirety of the test run (since all the buffered output is logged at once, at the end of the job).

I don't see how these limitations can be overcome for x86, but we can do better for arm.
Strategy:
 - copy android_emulator_unittest.py to androidx86_emulator_unittest.py
 - update buildbot config to use androidx86_emulator_unittest.py for x86 jobs
 - simplify android_emulator_unittest.py to support one job at a time
 - parse test output as it happens
Simple copy of android_emulator_unittest.py to androidx86_emulator_unittest.py.
Attachment #8605854 - Flags: review?(kmoir)
This simplifies android_emulator_unittest.py, stripping out the lists and loops used to support multiple emulators and multiple test suites. Test harness output is still buffered.
Attachment #8605859 - Flags: review?(kmoir)
This updates the way we run the test harness and report output. This is now similar to the way "normal" mozharness scripts work -- output is passed to the log parser as it happens.
See Also: → 1134349
Blocks: 1123828
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0eeb9fffb720

Notice the log timestamps...

http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-0eeb9fffb720/try-android-api-9/try_ubuntu64_vm_mobile_test-xpcshell-1-bm67-tests1-linux64-build56.txt.gz

09:57:39     INFO -  TEST-PASS | chrome/test/unit/test_abi.js | took 4465ms
09:57:39     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
09:57:40     INFO -  TEST-START | chrome/test/unit/test_bug292789.js
09:57:44     INFO -  TEST-PASS | chrome/test/unit/test_bug292789.js | took 3722ms
09:57:44     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
09:57:45     INFO -  TEST-START | chrome/test/unit/test_bug380398.js
09:57:49     INFO -  TEST-PASS | chrome/test/unit/test_bug380398.js | took 3762ms

Instead of the normal...

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-9/1431541543/mozilla-central_ubuntu64_vm_mobile_test-xpcshell-1-bm54-tests1-linux64-build29.txt.gz

13:33:26     INFO -  TEST-PASS | chrome/test/unit/test_abi.js | took 4346ms
13:33:26     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
13:33:26     INFO -  TEST-START | chrome/test/unit/test_bug292789.js
13:33:26     INFO -  TEST-PASS | chrome/test/unit/test_bug292789.js | took 3562ms
13:33:26     INFO -  mozdevice getting files in '/mnt/sdcard/tests/xpcshell/minidumps'
13:33:26     INFO -  TEST-START | chrome/test/unit/test_bug380398.js
13:33:26     INFO -  TEST-PASS | chrome/test/unit/test_bug380398.js | took 3631ms
Attachment #8605864 - Flags: review?(kmoir)
Attachment #8605854 - Flags: review?(kmoir) → review+
Attachment #8605855 - Flags: review?(kmoir) → review+
Attachment #8605859 - Flags: review?(kmoir) → review+
Attachment #8605864 - Flags: review?(kmoir) → review+
Keywords: leave-open
Before I can land the other patches here, particularly the buildbot config change, I need mozharness bumped to 560859d88ba8 (or later) on aurora, beta, and any other trees where Android x86 tests are run. :ryanvm, can you do this?
Flags: needinfo?(ryanvm)
https://hg.mozilla.org/releases/mozilla-esr38/rev/10f8fc9e6960

Doesn't run Android tests, but might as well keep them in sync if possible.
Flags: needinfo?(ryanvm)
Attachment #8605854 - Flags: checked-in+
Awesome. Thanks.
Attachment #8605855 - Flags: checked-in+
Keywords: leave-open
Attachment #8605859 - Flags: checked-in+
Attachment #8605864 - Flags: checked-in+
The last changes here started running on m-c 2015-05-22.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.