Android 2.3 (aws) content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | (various crashes and failures)

RESOLVED INCOMPLETE

Status

()

RESOLVED INCOMPLETE
5 years ago
10 months ago

People

(Reporter: gbrown, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [some tests disabled on Android 2.3])

Attachments

(2 attachments)

(Reporter)

Description

5 years ago
With the switch to aws, mochitest-gl has several failures and then reboots Android, and then retries...

2338 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/extensions/oes-vertex-array-object.html] Test page expected to fail, but all 24 tests passed
2453 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-abs.html] Test page expected to fail, but all 24 tests passed
2747 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-faceforward.html] Test page expected to fail, but all 24 tests passed
3111 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-sign.html] Test page expected to fail, but all 24 tests passed
3121 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-sin.html] Test failed - images are different
3137 INFO TEST-UNEXPECTED-FAIL | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-sin.html] (WebGL test error) 1 failure(s) and 0 timeout(s)
3163 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-step-float.html] Test page expected to fail, but all 24 tests passed
3189 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-step-gentype.html] Test page expected to fail, but all 24 tests passed
3215 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/glsl/functions/glsl-function-smoothstep-float.html] Test page expected to fail, but all 24 tests passed
3864 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/limits/gl-max-texture-dimensions.html] Test page expected to fail, but all 3 tests passed
3868 INFO TEST-UNEXPECTED-PASS | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/limits/gl-min-textures.html] Test page expected to fail, but all 2 tests passed
DMError: Remote Device Error: unable to connect to 127.0.0.1 after 5 attempts
Automation Error: Received unexpected exception while running application
Mochi-Remote ERROR | Automation Error: Exception caught while running tests
DMError: Remote Device Error: unable to connect to 127.0.0.1 after 5 attempts
(Reporter)

Updated

5 years ago
Blocks: 979921
(Reporter)

Updated

5 years ago
Keywords: intermittent-failure
(Reporter)

Comment 2

5 years ago
...but that did not help at all.

Now I notice:

19:06:42     INFO -  1 INFO TEST-START | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html
19:06:42     INFO -  2 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | GL vendor: Google (VMware, Inc.)
19:06:42     INFO -  3 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | GL renderer: Android Emulator OpenGL ES Translator (Gallium 0.4 on llvmpipe (LLVM 0x300))
19:06:42     INFO -  4 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | OS detected as: android
19:06:42     INFO -  5 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html |   Version: null
19:06:42     INFO -  6 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | GL driver detected as: mesa
19:06:42     INFO -  7 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | Failing tests file: failing_tests_android.txt
19:06:42     INFO -  8 INFO TEST-INFO | /tests/content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | Skipped tests file: skipped_tests_android.txt

On aws, llvmpipe is in the GL renderer string, and that over-rides the emulator logic, selecting the failing/skipped_tests_android.txt instead of the emulator versions.
(Reporter)

Comment 3

5 years ago
Second attempt -- force use of the _x86 (emulator) files that I was expecting: https://hg.mozilla.org/mozilla-central/rev/49c42fe2b315
(Reporter)

Comment 4

5 years ago
That failed again, but not as bad. I'll try skipping 2 more tests:

https://hg.mozilla.org/mozilla-central/rev/be076357691c
These need merging around to the integration branches and uplifted to Aurora.
Flags: needinfo?(cbook)
(In reply to Wes Kocher (:KWierso) from comment #5)
> These need merging around to the integration branches and uplifted to Aurora.

yeah but we wait for comment #4 also right ?
Flags: needinfo?(cbook)
(Reporter)

Comment 7

5 years ago
We are making progress, but the job is still perma-failing: https://tbpl.mozilla.org/php/getParsedLog.php?id=42621848&tree=Mozilla-Central&full=1.

I'll verify on try before my next attempt.
(In reply to Geoff Brown [:gbrown] from comment #7)
> We are making progress, but the job is still perma-failing:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=42621848&tree=Mozilla-
> Central&full=1.
> 
> I'll verify on try before my next attempt.

btw merged now whats was on m-c :) there are some gl crashes/issues on m-c i guess you are working on
(Reporter)

Updated

5 years ago
Summary: Android 2.3 (aws) content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | [conformance/extensions/oes-vertex-array-object.html] Test page expected to fail, but all 24 tests passed → Android 2.3 (aws) content/canvas/test/webgl-conformance/test_webgl_conformance_test_suite.html | (various crashes and failures)
Blocks: 1031364
(Reporter)

Comment 10

5 years ago
Skipping a few more tests finally gives a consistently green run:

https://tbpl.mozilla.org/?tree=Try&rev=cf9830a55cc4&showall=1

https://hg.mozilla.org/mozilla-central/rev/c90b38c47a1d

(I also removed entries from the failed list that are now skipped.)
(Reporter)

Comment 12

5 years ago
Created attachment 8447287 [details] [diff] [review]
recognize Android Emulator in preference to llvmpipe

:jgilbert -- This is a minor follow-up patch, but mostly I want to check in with you to explain what happened here and see if you want any other changes.

We introduced test infrastructure changes yesterday that moved Android 2.3 mochitests from ix slaves to aws. I accidentally failed to effectively test Android 2.3 mochitest-gl on aws before the production change. In production, crashes in mochitest-gl caused retries (blues on tbpl) which were not noticed until late in the day.

My initial attempt to skip the crashing tests failed because on aws, "llvmpipe" is in the GL renderer string, and that over-rides the emulator logic, selecting the failing/skipped_tests_android.txt files instead of the emulator versions. I hacked around that to select the emulator failing/skipped lists; this patch is an alternative. I don't feel strongly about either approach.

Even with the "correct" failing/skipped lists used, there were numerous failures, and unexpected passes, and crashes. Since the unexpected failures and passes do not happen on the Android x86 emulator, I felt I had to skip those. So in the end I added these to skipped_tests_android_x86.txt:

 +conformance/extensions/oes-vertex-array-object.html
 +conformance/glsl/functions/glsl-function-abs.html
 +conformance/glsl/functions/glsl-function-faceforward.html
 +conformance/glsl/functions/glsl-function-sign.html
 +conformance/glsl/functions/glsl-function-sin.html
 +conformance/glsl/functions/glsl-function-step-float.html
 +conformance/glsl/functions/glsl-function-step-gentype.html
 +conformance/glsl/functions/glsl-function-smoothstep-float.html
 +conformance/limits/gl-max-texture-dimensions.html
 +conformance/limits/gl-min-textures.html
 +conformance/misc/error-reporting.html
 +conformance/misc/object-deletion-behaviour.html
 +conformance/misc/type-conversion-test.html
 +conformance/reading/read-pixels-test.html
 +conformance/renderbuffers/framebuffer-object-attachment.html
 +conformance/textures/gl-teximage.html
 +conformance/textures/origin-clean-conformance.html
 +conformance/textures/tex-image-and-sub-image-2d-with-image.html
 +conformance/textures/tex-image-and-sub-image-2d-with-array-buffer-view.html
 +conformance/textures/texture-mips.html
 +conformance/textures/texture-size.html
 +conformance/textures/texture-size-cube-maps.html

This puts the list of skipped Android emulator tests back to a state similar to what it was before my recent cleanup in bug 1020642. :)
Attachment #8447287 - Flags: review?(jgilbert)
(Reporter)

Updated

5 years ago
Keywords: intermittent-failure → leave-open
Whiteboard: [some tests disabled on Android 2.3]
(Reporter)

Updated

5 years ago
Component: General → Canvas: WebGL
Product: Firefox for Android → Core
Attachment #8447287 - Flags: review?(jgilbert) → review+
(Reporter)

Comment 15

5 years ago
Leaving open for follow-up on disabled tests as/if needed.
Assignee: gbrown → nobody
(Reporter)

Comment 16

3 years ago
Created attachment 8734586 [details] [diff] [review]
update manifests for android mochitest-gl

The minimum Android api level supported by Firefox 48 is 15. Android 2.3 and 4.0 tests no longer run on trunk. This provides an opportunity to clean up Android skip and fail annotations significantly. I tried removing all of the Android annotations, then added back those that caused failures on Android 4.3 until I got a clean try run:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=25d507df4e51

I made one hand-edit to _webgl-conformance.ini: generate-wrappers-and-manifest.py added e10s to the skip-if in the DEFAULT section; this seemed unrelated to my changes, and, unsure of the history here, I was reluctant to introduce such a change, so I undid that change to the DEFAULT skip-if.
Attachment #8734586 - Flags: review?(jgilbert)
(Reporter)

Updated

3 years ago
See Also: → bug 1257948
Comment on attachment 8734586 [details] [diff] [review]
update manifests for android mochitest-gl

Review of attachment 8734586 [details] [diff] [review]:
-----------------------------------------------------------------

We're landing 1.0.3 conformance tests soon (bug 1193526), so this should wait until after that.
Attachment #8734586 - Flags: review?(jgilbert)
Depends on: 1193526

Comment 18

a year ago
Android 2 is no longer supported.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → INCOMPLETE
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.