1031097 - Android 2.3 (aws) content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | (various crashes and failures)

Status: RESOLVED INCOMPLETE

(Core :: Graphics: CanvasWebGL, defect)

Description

[Geoff Brown [:gbrown]]

With the switch to aws, mochitest-gl has several failures and then reboots Android, and then retries...

2338 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/extensions/oes-vertex-array-object.html] Test page expected to fail, but all 24 tests passed
2453 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-abs.html] Test page expected to fail, but all 24 tests passed
2747 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-faceforward.html] Test page expected to fail, but all 24 tests passed
3111 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-sign.html] Test page expected to fail, but all 24 tests passed
3121 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-sin.html] Test failed - images are different
3137 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-sin.html] (WebGL test error) 1 failure(s) and 0 timeout(s)
3163 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-step-float.html] Test page expected to fail, but all 24 tests passed
3189 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-step-gentype.html] Test page expected to fail, but all 24 tests passed
3215 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-smoothstep-float.html] Test page expected to fail, but all 24 tests passed
3864 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/limits/gl-max-texture-dimensions.html] Test page expected to fail, but all 3 tests passed
3868 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/limits/gl-min-textures.html] Test page expected to fail, but all 2 tests passed
DMError: Remote Device Error: unable to connect to 127.0.0.1 after 5 attempts
Automation Error: Received unexpected exception while running application
Mochi-Remote ERROR | Automation Error: Exception caught while running tests
DMError: Remote Device Error: unable to connect to 127.0.0.1 after 5 attempts

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Geoff pushed https://hg.mozilla.org/mozilla-central/rev/6c1da87675d9 to m-c to try to fix this, and I pushed it to Aurora as https://hg.mozilla.org/releases/mozilla-aurora/rev/a101306a4aa6

Comment 2

[Geoff Brown [:gbrown]]

...but that did not help at all.

Now I notice:

19:06:42     INFO -  1 INFO TEST-START | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html
19:06:42     INFO -  2 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | GL vendor: Google (VMware, Inc.)
19:06:42     INFO -  3 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | GL renderer: Android Emulator OpenGL ES Translator (Gallium 0.4 on llvmpipe (LLVM 0x300))
19:06:42     INFO -  4 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | OS detected as: android
19:06:42     INFO -  5 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html |   Version: null
19:06:42     INFO -  6 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | GL driver detected as: mesa
19:06:42     INFO -  7 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | Failing tests file: failing_tests_android.txt
19:06:42     INFO -  8 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | Skipped tests file: skipped_tests_android.txt

On aws, llvmpipe is in the GL renderer string, and that over-rides the emulator logic, selecting the failing/skipped_tests_android.txt instead of the emulator versions.

Comment 3

[Geoff Brown [:gbrown]]

Second attempt -- force use of the _x86 (emulator) files that I was expecting: https://hg.mozilla.org/mozilla-central/rev/49c42fe2b315

Comment 4

[Geoff Brown [:gbrown]]

That failed again, but not as bad. I'll try skipping 2 more tests:

https://hg.mozilla.org/mozilla-central/rev/be076357691c

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

These need merging around to the integration branches and uplifted to Aurora.

Comment 6

[Carsten Book [:Tomcat]]

(In reply to Wes Kocher (:KWierso) from comment #5)
> These need merging around to the integration branches and uplifted to Aurora.

yeah but we wait for comment #4 also right ?

Comment 7

[Geoff Brown [:gbrown]]

We are making progress, but the job is still perma-failing: https://tbpl.mozilla.org/php/getParsedLog.php?id=42621848&tree=Mozilla-Central&full=1.

I'll verify on try before my next attempt.

Comment 8

[Carsten Book [:Tomcat]]

(In reply to Geoff Brown [:gbrown] from comment #7)
> We are making progress, but the job is still perma-failing:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=42621848&tree=Mozilla-
> Central&full=1.
> 
> I'll verify on try before my next attempt.

btw merged now whats was on m-c :) there are some gl crashes/issues on m-c i guess you are working on

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/c996b80f1ec1
https://hg.mozilla.org/releases/mozilla-aurora/rev/f6ed824e0778

Comment 10

[Geoff Brown [:gbrown]]

Skipping a few more tests finally gives a consistently green run:

https://tbpl.mozilla.org/?tree=Try&rev=cf9830a55cc4&showall=1

https://hg.mozilla.org/mozilla-central/rev/c90b38c47a1d

(I also removed entries from the failed list that are now skipped.)

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/4d2d5b361135

Comment 12

[Geoff Brown [:gbrown]]

Attachment: recognize Android Emulator in preference to llvmpipe

:jgilbert -- This is a minor follow-up patch, but mostly I want to check in with you to explain what happened here and see if you want any other changes.

We introduced test infrastructure changes yesterday that moved Android 2.3 mochitests from ix slaves to aws. I accidentally failed to effectively test Android 2.3 mochitest-gl on aws before the production change. In production, crashes in mochitest-gl caused retries (blues on tbpl) which were not noticed until late in the day.

My initial attempt to skip the crashing tests failed because on aws, "llvmpipe" is in the GL renderer string, and that over-rides the emulator logic, selecting the failing/skipped_tests_android.txt files instead of the emulator versions. I hacked around that to select the emulator failing/skipped lists; this patch is an alternative. I don't feel strongly about either approach.

Even with the "correct" failing/skipped lists used, there were numerous failures, and unexpected passes, and crashes. Since the unexpected failures and passes do not happen on the Android x86 emulator, I felt I had to skip those. So in the end I added these to skipped_tests_android_x86.txt:

 +conformance/extensions/oes-vertex-array-object.html
 +conformance/glsl/functions/glsl-function-abs.html
 +conformance/glsl/functions/glsl-function-faceforward.html
 +conformance/glsl/functions/glsl-function-sign.html
 +conformance/glsl/functions/glsl-function-sin.html
 +conformance/glsl/functions/glsl-function-step-float.html
 +conformance/glsl/functions/glsl-function-step-gentype.html
 +conformance/glsl/functions/glsl-function-smoothstep-float.html
 +conformance/limits/gl-max-texture-dimensions.html
 +conformance/limits/gl-min-textures.html
 +conformance/misc/error-reporting.html
 +conformance/misc/object-deletion-behaviour.html
 +conformance/misc/type-conversion-test.html
 +conformance/reading/read-pixels-test.html
 +conformance/renderbuffers/framebuffer-object-attachment.html
 +conformance/textures/gl-teximage.html
 +conformance/textures/origin-clean-conformance.html
 +conformance/textures/tex-image-and-sub-image-2d-with-image.html
 +conformance/textures/tex-image-and-sub-image-2d-with-array-buffer-view.html
 +conformance/textures/texture-mips.html
 +conformance/textures/texture-size.html
 +conformance/textures/texture-size-cube-maps.html

This puts the list of skipped Android emulator tests back to a state similar to what it was before my recent cleanup in bug 1020642. :)

Comment 13

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/716cecc3c0b4

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/716cecc3c0b4

Comment 15

[Geoff Brown [:gbrown]]

Leaving open for follow-up on disabled tests as/if needed.

Comment 16

[Geoff Brown [:gbrown]]

Attachment: update manifests for android mochitest-gl

The minimum Android api level supported by Firefox 48 is 15. Android 2.3 and 4.0 tests no longer run on trunk. This provides an opportunity to clean up Android skip and fail annotations significantly. I tried removing all of the Android annotations, then added back those that caused failures on Android 4.3 until I got a clean try run:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=25d507df4e51

I made one hand-edit to _webgl-conformance.ini: generate-wrappers-and-manifest.py added e10s to the skip-if in the DEFAULT section; this seemed unrelated to my changes, and, unsure of the history here, I was reluctant to introduce such a change, so I undid that change to the DEFAULT skip-if.

Comment 17

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8734586 [details] [diff] [review]
update manifests for android mochitest-gl

Review of attachment 8734586 [details] [diff] [review]:
-----------------------------------------------------------------

We're landing 1.0.3 conformance tests soon (bug 1193526), so this should wait until after that.

Comment 18

[u597032]

Android 2 is no longer supported.

Comment 19

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.