Closed Bug 1140148 Opened 5 years ago Closed 5 years ago

"Green-up" Android 4.3 Opt emulator tests

Categories

(Testing :: General, defect)

x86_64
Linux
defect
Not set

Tracking

(firefox39 fixed)

RESOLVED FIXED
mozilla39
Tracking Status
firefox39 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(5 files, 6 obsolete files)

We want to run most of our test suites on an emulator running Android 4.3, as developed in bug 1062365. Of course some tests fail in that environment; this bug will update manifests to mark tests as failing or skip them on Android 4.3.
Attached patch wip - xpcshell manifest updates (obsolete) — Splinter Review
Attached patch wip - reftest manifests (obsolete) — Splinter Review
Attached patch wip - robocop manifest (obsolete) — Splinter Review
Attached patch wip - mochitest-gl manifests (obsolete) — Splinter Review
Attached patch wip - mochitest manifests (obsolete) — Splinter Review
"TODO: NEW BUG"

I <3 you so much.
Blocks: 1140454
Attachment #8573548 - Attachment is obsolete: true
Attachment #8579050 - Flags: review?(dminor)
Attachment #8573555 - Attachment is obsolete: true
Attachment #8579061 - Flags: review?(jgilbert)
Comment on attachment 8579050 [details] [diff] [review]
xpcshell manifest updates for Android 4.3

Review of attachment 8579050 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm!
Attachment #8579050 - Flags: review?(dminor) → review+
Attachment #8579061 - Flags: review?(jgilbert) → review+
mochitest-gl changes for 2.3 will apply to 4.3 too. Synced up the configs again:

https://hg.mozilla.org/build/mozharness/rev/d99684a19ced
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #11)
> mochitest-gl patch backed out for mochitest-1 and mochitest-gl bustage.

There is a syntax error in that patch:

03-20 16:46:04.789 W/GeckoConsole( 2258): [JavaScript Error: "ReferenceError: invalid assignment left-hand side" {file: "http://mochi.test:8888/tests/dom/canvas/test/webgl-conformance/mochi-single.html?conformance/attribs/gl-enable-vertex-attrib.html" line: 124 column: 16 source: "        failSet['conformance/extensions/oes-texture-float.html' = true;
03-20 16:46:04.789 W/GeckoConsole( 2258): "}]

(missing trailing ']').
Attachment #8573551 - Attachment is obsolete: true
Attachment #8581714 - Flags: review?(snorp)
Comment on attachment 8581714 [details] [diff] [review]
robocop.ini updates for android 4.3

Review of attachment 8581714 [details] [diff] [review]:
-----------------------------------------------------------------

Shouldn't it really be "skip-if android_emulator"?
Certainly the majority of robocop annotations seem to apply to both 2.3 and 4.3. It's interesting (to me anyway!) that many of the tests fail differently on 2.3 than on 4.3 -- different failures, different bugs, different causes? Or just poor tests, susceptible to timing variations?

We do not currently have a way for the harness to detect if the tests are running on emulator. I suppose we could introduce an alias mechanism so "android_emulator" expands to "android_version == 11 || android_version == 18" but I'm not sure that would be a significant improvement.
Attachment #8581714 - Flags: review?(snorp) → review+
Attachment #8573550 - Attachment is obsolete: true
Attachment #8582082 - Flags: review?(dminor)
Attachment #8582082 - Flags: review?(dminor) → review+
Attached patch wip - mochitest manifests (obsolete) — Splinter Review
Attachment #8573556 - Attachment is obsolete: true
Blocks: 1146574
Attachment #8583391 - Attachment is obsolete: true
Attachment #8583984 - Flags: review?(dminor)
Attachment #8583984 - Flags: review?(dminor) → review+
Keywords: leave-open
Summary: "Green-up" Android 4.3 emulator tests → "Green-up" Android 4.3 Opt emulator tests
Should there be 4.3 jobs on that run?
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #28)
> Should there be 4.3 jobs on that run?

Ha! There are! The "Android 2.3 API9 opt" tests in that try run are actually using the build from "Android 4.0 API11+ opt" and running with the Android 4.3 mozharness configuration, which runs Android 4.3 in the emulator. How? https://hg.mozilla.org/try/rev/92c69392f2a2 points mozharness to my user repo, which has hacks to replace the apk and tests.zip urls and modify the configuration file name.

As usual, the logs tell the story:

http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/gbrown@mozilla.com-92c69392f2a2/try-android-api-9/try_ubuntu64_vm_mobile_test-xpcshell-1-bm121-tests1-linux64-build91.txt.gz

16:59:54     INFO - Run as scripts/scripts/android_emulator_unittest.py --cfg android/androidarm_4_3.py --test-suite xpcshell-1 --blob-upload-branch try --download-symbols ondemand
...
17:00:00     INFO -  INFO - File AVDs-armv7a-android-4.3.1_r1-build-2015-03-16.tar.gz fetched from http://tooltool.pvt.build.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/build/ as /home/cltbld/.android/tmp7iIFYw
...
17:01:23     INFO - Downloading http://ftp.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/pub/mozilla.org/firefox/try-builds/gbrown@mozilla.com-92c69392f2a2/try-android-api-11/fennec-39.0a1.en-US.android-arm.tests.zip to /builds/slave/test/build/fennec-39.0a1.en-US.android-arm.tests.zip
...
17:02:07     INFO - Downloading http://ftp.mozilla.org.proxxy1.srv.releng.use1.mozilla.com/pub/mozilla.org/firefox/try-builds/gbrown@mozilla.com-92c69392f2a2/try-android-api-11/fennec-39.0a1.en-US.android-arm.apk to /builds/slave/test/build/fennec-39.0a1.en-US.android-arm.apk
https://hg.mozilla.org/mozilla-central/rev/6d9dd4432988
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla39
You need to log in before you can comment on or make changes to this bug.