Fix the WebGL tests on Android x86 emulator test slaves

RESOLVED FIXED in mozilla27

Status

()

defect
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: gbrown, Assigned: gbrown)

Tracking

Trunk
mozilla27
x86
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

+++ This bug was initially created as a clone of Bug #874291 +++

In the Android x86 emulator test environment we have slightly different test failures than in the normal Android case: https://tbpl.mozilla.org/php/getParsedLog.php?id=27849762&tree=Cedar&full=1#error0

I'd like to adjust our failure expectations for this new environment.
Posted patch wip (obsolete) — Splinter Review
Blocks: 891959
No longer blocks: 895186
With these changes, I get a clean run:

13 INFO TEST-START | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
14 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | GL vendor: Google (NVIDIA Corporation)
15 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | GL renderer: Android Emulator OpenGL ES Translator (GeForce GT 610/PCIe/SSE2)
16 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | OS detected as: android
17 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html |   Version: null
18 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | GL driver detected as: android x86 emulator
19 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Failing tests file: failing_tests     _android_x86.txt
20 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Skipped tests file: skipped_tests     _android.txt
21 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Tests to fail: 7:
22 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html |   conformance/extensions/oes-text     ure-float.html
...
29 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Tests to skip: 39:
30 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html |   conformance/glsl/functions/glsl-function-abs.html
...
7729 INFO TEST-END | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | finished in 622368ms
7730 INFO TEST-START | Shutdown
7731 INFO Passed:  7026
7732 INFO Failed:  0
7733 INFO Todo:    308
7734 INFO Slowest: 4044ms - /tests/content/canvas/test/webgl/non-conf-tests/test_webgl_conformance.html
7735 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:11:25.896094
Attachment #805681 - Attachment is obsolete: true
Attachment #806192 - Flags: review?(bjacob)
Comment on attachment 806192 [details] [diff] [review]
use new manifests for android x86

Review of attachment 806192 [details] [diff] [review]:
-----------------------------------------------------------------

This is a lot of tests to skip! Are they really all crashy?
(In reply to Jeff Gilbert [:jgilbert] from comment #3)
> This is a lot of tests to skip! Are they really all crashy?

Most of these tests are skipped on Android already. I just added:

conformance/textures/tex-image-and-sub-image-2d-with-video.html
conformance/textures/texture-mips.html
conformance/textures/texture-npot-video.html

I did not verify that all of the existing android skipped tests also crash on the x86 emulator. (I hesitate to change those since they could be intermittent failures/crashes.)
conformance/textures/texture-mips.html is not a crash but an intermittent failure. The other two (video) tests both crash with libstagefright in the stack, similar to bug 914439.
(In reply to Geoff Brown [:gbrown] from comment #4)
> I did not verify that all of the existing android skipped tests also crash
> on the x86 emulator. (I hesitate to change those since they could be
> intermittent failures/crashes.)

I checked and it looks like the list of skipped test for android x86 can be reduced to:

  conformance/textures/tex-image-and-sub-image-2d-with-video.html
  conformance/textures/texture-mips.html
  conformance/textures/texture-npot-video.html
  conformance/textures/texture-size.html

I have only run 10 times with this reduced list, so there could be intermittents, and I don't know the history of the original android skip list -- not sure if I should keep the list in the patch up for review, or reduce it to this set.
I found 100 runs more satisfactory, the last time I was dealing with these lists (but then, I started with empty lists and did several cycles of 100 to build them back up as I got rid of crashes, so my total was probably more like 1000).
Attachment #806192 - Flags: review?(bjacob) → review+
https://hg.mozilla.org/mozilla-central/rev/b7bc1f794012
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla27
Blocks: 936226
You need to log in before you can comment on or make changes to this bug.