917053 - Fix the WebGL tests on Android x86 emulator test slaves

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect)

Description

[Geoff Brown [:gbrown]]

+++ This bug was initially created as a clone of Bug #874291 +++

In the Android x86 emulator test environment we have slightly different test failures than in the normal Android case: https://tbpl.mozilla.org/php/getParsedLog.php?id=27849762&tree=Cedar&full=1#error0

I'd like to adjust our failure expectations for this new environment.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: wip

Comment 2

[Geoff Brown [:gbrown]]

Attachment: use new manifests for android x86

With these changes, I get a clean run:

13 INFO TEST-START | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html
14 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | GL vendor: Google (NVIDIA Corporation)
15 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | GL renderer: Android Emulator OpenGL ES Translator (GeForce GT 610/PCIe/SSE2)
16 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | OS detected as: android
17 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html |   Version: null
18 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | GL driver detected as: android x86 emulator
19 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Failing tests file: failing_tests     _android_x86.txt
20 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Skipped tests file: skipped_tests     _android.txt
21 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Tests to fail: 7:
22 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html |   conformance/extensions/oes-text     ure-float.html
...
29 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | Tests to skip: 39:
30 INFO TEST-INFO | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html |   conformance/glsl/functions/glsl-function-abs.html
...
7729 INFO TEST-END | /tests/content/canvas/test/webgl/test_webgl_conformance_test_suite.html | finished in 622368ms
7730 INFO TEST-START | Shutdown
7731 INFO Passed:  7026
7732 INFO Failed:  0
7733 INFO Todo:    308
7734 INFO Slowest: 4044ms - /tests/content/canvas/test/webgl/non-conf-tests/test_webgl_conformance.html
7735 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:11:25.896094

Comment 3

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 806192 [details] [diff] [review]
use new manifests for android x86

Review of attachment 806192 [details] [diff] [review]:
-----------------------------------------------------------------

This is a lot of tests to skip! Are they really all crashy?

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Jeff Gilbert [:jgilbert] from comment #3)
> This is a lot of tests to skip! Are they really all crashy?

Most of these tests are skipped on Android already. I just added:

conformance/textures/tex-image-and-sub-image-2d-with-video.html
conformance/textures/texture-mips.html
conformance/textures/texture-npot-video.html

I did not verify that all of the existing android skipped tests also crash on the x86 emulator. (I hesitate to change those since they could be intermittent failures/crashes.)

Comment 5

[Geoff Brown [:gbrown]]

conformance/textures/texture-mips.html is not a crash but an intermittent failure. The other two (video) tests both crash with libstagefright in the stack, similar to bug 914439.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #4)
> I did not verify that all of the existing android skipped tests also crash
> on the x86 emulator. (I hesitate to change those since they could be
> intermittent failures/crashes.)

I checked and it looks like the list of skipped test for android x86 can be reduced to:

  conformance/textures/tex-image-and-sub-image-2d-with-video.html
  conformance/textures/texture-mips.html
  conformance/textures/texture-npot-video.html
  conformance/textures/texture-size.html

I have only run 10 times with this reduced list, so there could be intermittents, and I don't know the history of the original android skip list -- not sure if I should keep the list in the patch up for review, or reduce it to this set.

Comment 7

[Phil Ringnalda (:philor)]

I found 100 runs more satisfactory, the last time I was dealing with these lists (but then, I started with empty lists and did several cycles of 100 to build them back up as I got rid of crashes, so my total was probably more like 1000).

Comment 8

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b7bc1f794012

Comment 9

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/b7bc1f794012