1020642 - Review failed/skipped webgl-conformance tests for Android

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

content/canvas/test/webgl-conformance contains platform-specific text files for failed and skipped webgl-conformance tests. 

There are variants for "android" and "android_x86". Android 2.3 mochitest-gl accidentally uses "android_x86"; it should probably have its own variant, or use "android". 

Some "android" tests may have been added for Android 2.2 and may be okay on Android 4.0 -- let's review which entries are really needed.

Some tests are marked as both skipped and failing for "android" -- they should probably just be skipped.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: remove many entries from Android webgl-conformance failing/skipped lists

Using multiple try runs, I found that many more webgl-conformance tests can be run on Android without triggering test failures -- reflected in these updated lists.

Note that the new entries added to the failed tests lists were previously skipped (they don't crash and seem to fail reliably).

When I reported this bug, I noted that the Android 2.3 Opt and Android 4.2 x86 Opt tests both use the same failed/skipped_tests_android_x86.txt lists. That's because glVendor and glRenderer are identical for these platforms (both run in an emulator). We could differentiate based on kOSVersion, or perhaps some more appropriate property, but it looks like that would only allow a couple of additional tests to be run -- I don't think it warrants a change.

Comment 2

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8444002 [details] [diff] [review]
remove many entries from Android webgl-conformance failing/skipped lists

Review of attachment 8444002 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thank you!

Comment 3

[Kelsey Gilbert [:jgilbert]]

(In reply to Geoff Brown [:gbrown] from comment #1)
> When I reported this bug, I noted that the Android 2.3 Opt and Android 4.2
> x86 Opt tests both use the same failed/skipped_tests_android_x86.txt lists.
> That's because glVendor and glRenderer are identical for these platforms
> (both run in an emulator). We could differentiate based on kOSVersion, or
> perhaps some more appropriate property, but it looks like that would only
> allow a couple of additional tests to be run -- I don't think it warrants a
> change.

Yep. Ideally, we want to be able to set failed/skipped tests by slave configuration.

Comment 4

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ca69915acaf3

Comment 5

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/ca69915acaf3

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/cc9cbb67d06d