1193526 - Upgrade WebGL conformance tests to 1.0.3

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect)

Description

[Kyle Fung]

Upgrade the WebGL conformance tests being run to 1.0.3, since we are currently a bit far from total conformance and don't want to regress.

Comment 1

[Kyle Fung]

Attachment: 001-nuke-dir.patch

WIP

Comment 2

[Kyle Fung]

Attachment: 002-manifest-ver-case.patch

Comment 3

[Kyle Fung]

Attachment: 003-report-fail.patch

Comment 4

[Kyle Fung]

Attachment: 004-dont-load-internet.patch

Comment 5

[Kyle Fung]

Attachment: 005-more-info.patch

Tests are running and seem to be reporting errors properly on try. Trying to compile an errata that doesn't crash/fail for Linux opt builds.

Comment 6

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8650546 [details] [diff] [review]
001-nuke-dir.patch

Review of attachment 8650546 [details] [diff] [review]:
-----------------------------------------------------------------

This looks right.

Comment 7

[Kyle Fung]

Attachment: 006-shorten-path-name.patch

Pathname of a certain test file is too long for the Windows machines to handle, so it was shortened with this patch.

Comment 8

[Kyle Fung]

Attachment: 007-omit-crashed-tests.patch

This patch omits tests that crash while running.

Comment 9

[Kyle Fung]

Attachment: 008-omit-ogles-tests.patch

This patch modifies the test manifests to skip the entire ogles section of tests since they seem to randomly cause later tests to fail.

However, the tests will still crash on Linux 64 bit debug builds seemingly randomly:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=414cd8d1c104

Since the 1.0.1 tests don't seem to have a ogles section, I think it's okay to omit them for now.

The random crashes in Linux 64 seem to be caused by certain tests in the glsl section of tests since the tests don't crash when both the ogles and glsl tests are disabled:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=927db8e97b8f

Seeing as the glsl tests are present in the 1.0.1 version of the suite, we probably shouldn't omit them and find the subset of glsl tests that cause the crashes.

Comment 10

[Jeff Muizelaar [:jrmuizel]]

(In reply to Kyle Fung from comment #9)
> 
> Seeing as the glsl tests are present in the 1.0.1 version of the suite, we
> probably shouldn't omit them and find the subset of glsl tests that cause
> the crashes.

To make things easy we could just omit the new glsl tests for now.

Comment 11

[Jeff Muizelaar [:jrmuizel]]

Here's an attempt at that:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3bbe1ce3c8e6

Comment 12

[Jeff Muizelaar [:jrmuizel]]

(In reply to Kyle Fung from comment #9)
> The random crashes in Linux 64 seem to be caused by certain tests in the
> glsl section of tests since the tests don't crash when both the ogles and
> glsl tests are disabled:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=927db8e97b8f

I retriggered the job a couple of times and got a crash in the driver.

Comment 13

[Kyle Fung]

Is the crash intermittence of your push too frequent to be acceptable?
I've been observing frequencies of around that as well, but I figured that they were infrequent enough to be ignored.

Comment 14

[Jeff Muizelaar [:jrmuizel]]

(In reply to Kyle Fung from comment #13)
> Is the crash intermittence of your push too frequent to be acceptable?
> I've been observing frequencies of around that as well, but I figured that
> they were infrequent enough to be ignored.

We don't want to knowingly introduce an intermittent. I don't think the current tests crash intermittently, so this warrants further investigation.

Comment 15

[Kelsey Gilbert [:jgilbert]]

I'm going to take this bug back. We're basically ready-to-go here with the exception of the Mesa software driver update.

Comment 16

[Kelsey Gilbert [:jgilbert]]

This doesn't require passing the tests. I have created bug 1246063 for tracking general 1.0.3 failures.

Comment 17

[Kelsey Gilbert [:jgilbert]]

Dropping the alias, and see-also'ing bugs for both 103 failures and 103 conformance.

Comment 19

[Kelsey Gilbert [:jgilbert]]

It's at the point where we're just needing to nail down the remaining failures, particularly the troublesome Android frequent oranges:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=244b3c957fd0

Here's my WIP branch:
https://github.com/jdashg/gecko-dev/tree/test-103-2

Comment 20

[Milan Sreckovic [:milan] (needinfo for best results)]

Peter, this is our next step in the WebGL2 work; anyone available on your side?

Comment 21

[Peter Chang[:pchang]]

(In reply to Milan Sreckovic [:milan] from comment #20)
> Peter, this is our next step in the WebGL2 work; anyone available on your
> side?

Ethan will be the main owner for this. And it would be great to use etherpad(or bugzilla) to track these failures and others can know which one they can work on if they are available.

Comment 22

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

I can handle this.

Comment 24

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Remove old conformance tests and generated files. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48367/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48367/

Comment 25

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Update conformance path and remove unused file. r=jgilbert

The folder name "webgl-conformance" is too long. Rename it to
"webgl-conf".

Review commit: https://reviewboard.mozilla.org/r/48369/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48369/

Comment 26

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Update all generator files for webgl conformance test 1.0.3. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48371/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48371/

Comment 27

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Update mochitest paths. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48373/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48373/

Comment 28

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Update tests to 1.0.3 from a8ffce2eb6906f3b910399de9ce82dc5538df95b. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48375/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48375/

Comment 29

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Rename built_in_varying_array_out_of_bounds to biv_array_oob. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48377/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48377/

Comment 30

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Handle video failing to load. r=jgilbert

The name of some tests is too long and exceed maximum length of windows
path. So we rename it to shorter name.

Review commit: https://reviewboard.mozilla.org/r/48379/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48379/

Comment 31

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Handle image failing to load. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48381/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48381/

Comment 32

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Handle image.src being invalid. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48383/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48383/

Comment 33

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Update errata to mark all failures in 1.0.3 webgl conformance tests. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48385/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48385/

Comment 34

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Add generated files. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48387/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48387/

Comment 35

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Attachment: MozReview Request: Bug 1193526 - Add generated files. r=jgilbert

Review commit: https://reviewboard.mozilla.org/r/48389/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48389/

Comment 36

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1a6755c0efb8

Comment 37

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744182 [details]
MozReview Request: Bug 1193526 - Remove old conformance tests and generated files. r=jgilbert

https://reviewboard.mozilla.org/r/48367/#review45291

Comment 38

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744183 [details]
MozReview Request: Bug 1193526 - Update conformance path and remove unused file. r=jgilbert

https://reviewboard.mozilla.org/r/48369/#review45293

Comment 39

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744184 [details]
MozReview Request: Bug 1193526 - Update all generator files for webgl conformance test 1.0.3. r=jgilbert

https://reviewboard.mozilla.org/r/48371/#review45295

Comment 40

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744185 [details]
MozReview Request: Bug 1193526 - Update mochitest paths. r=jgilbert

https://reviewboard.mozilla.org/r/48373/#review45297

Comment 41

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744186 [details]
MozReview Request: Bug 1193526 - Update tests to 1.0.3 from a8ffce2eb6906f3b910399de9ce82dc5538df95b. r=jgilbert

https://reviewboard.mozilla.org/r/48375/#review45299

Comment 42

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744188 [details]
MozReview Request: Bug 1193526 - Handle video failing to load. r=jgilbert

https://reviewboard.mozilla.org/r/48379/#review45301

Comment 43

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744193 [details]
MozReview Request: Bug 1193526 - Add generated files. r=jgilbert

https://reviewboard.mozilla.org/r/48389/#review45305

Comment 44

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744191 [details]
MozReview Request: Bug 1193526 - Update errata to mark all failures in 1.0.3 webgl conformance tests. r=jgilbert

https://reviewboard.mozilla.org/r/48385/#review45307

Comment 45

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744190 [details]
MozReview Request: Bug 1193526 - Handle image.src being invalid. r=jgilbert

https://reviewboard.mozilla.org/r/48383/#review45309

Comment 46

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744189 [details]
MozReview Request: Bug 1193526 - Handle image failing to load. r=jgilbert

https://reviewboard.mozilla.org/r/48381/#review45311

Comment 47

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744192 [details]
MozReview Request: Bug 1193526 - Add generated files. r=jgilbert

https://reviewboard.mozilla.org/r/48387/#review45303

::: dom/canvas/test/webgl-mochitest/mochitest.ini:86
(Diff revision 1)
>  [test_webgl_conformance.html]
>  skip-if = toolkit == 'android' #bug 865443- seperate suite - the non_conf* tests pass except for one on armv6 tests
>  [test_webgl_compressed_texture_es3.html]
>  [test_webgl_disjoint_timer_query.html]
>  [test_webgl_force_enable.html]
> +skip-if = (os == 'android')

Woah, why is this here?
We really want to keep this!

Comment 48

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744187 [details]
MozReview Request: Bug 1193526 - Rename built_in_varying_array_out_of_bounds to biv_array_oob. r=jgilbert

https://reviewboard.mozilla.org/r/48377/#review45313

It looks like this reverts the file. We shouldn't revert this file. We should try to work with the file as-is, and only make minor changes (and upstream them) if we need to.

Comment 49

[Kelsey Gilbert [:jgilbert]]

https://reviewboard.mozilla.org/r/48377/#review45315

::: dom/canvas/test/webgl-conf/checkout/resources/webgl-test-harness.js:1
(Diff revision 1)
> -/*
> +// Copyright (c) 2009 The Chromium Authors. All rights reserved.

Let's try to use the current version of this file.

Comment 50

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744182 [details]
MozReview Request: Bug 1193526 - Remove old conformance tests and generated files. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48367/diff/1-2/

Comment 51

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744183 [details]
MozReview Request: Bug 1193526 - Update conformance path and remove unused file. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48369/diff/1-2/

Comment 52

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744184 [details]
MozReview Request: Bug 1193526 - Update all generator files for webgl conformance test 1.0.3. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48371/diff/1-2/

Comment 53

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744185 [details]
MozReview Request: Bug 1193526 - Update mochitest paths. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48373/diff/1-2/

Comment 54

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744186 [details]
MozReview Request: Bug 1193526 - Update tests to 1.0.3 from a8ffce2eb6906f3b910399de9ce82dc5538df95b. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48375/diff/1-2/

Comment 55

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744187 [details]
MozReview Request: Bug 1193526 - Rename built_in_varying_array_out_of_bounds to biv_array_oob. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48377/diff/1-2/

Comment 56

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744188 [details]
MozReview Request: Bug 1193526 - Handle video failing to load. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48379/diff/1-2/

Comment 57

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744189 [details]
MozReview Request: Bug 1193526 - Handle image failing to load. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48381/diff/1-2/

Comment 58

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744190 [details]
MozReview Request: Bug 1193526 - Handle image.src being invalid. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48383/diff/1-2/

Comment 59

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744191 [details]
MozReview Request: Bug 1193526 - Update errata to mark all failures in 1.0.3 webgl conformance tests. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48385/diff/1-2/

Comment 60

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744192 [details]
MozReview Request: Bug 1193526 - Add generated files. r=jgilbert

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/48387/diff/1-2/

Comment 61

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Because I removed one part of patch, review board got confused......

Comment 62

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744187 [details]
MozReview Request: Bug 1193526 - Rename built_in_varying_array_out_of_bounds to biv_array_oob. r=jgilbert

This part should be r+. jgilbert, can you set r+ in review board?

Comment 63

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744192 [details]
MozReview Request: Bug 1193526 - Add generated files. r=jgilbert

This part should be r+ as well...

Comment 64

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Comment on attachment 8744191 [details]
MozReview Request: Bug 1193526 - Update errata to mark all failures in 1.0.3 webgl conformance tests. r=jgilbert

This is the exactly one should be reviewed.

Comment 65

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1213328f8626

Comment 66

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744187 [details]
MozReview Request: Bug 1193526 - Rename built_in_varying_array_out_of_bounds to biv_array_oob. r=jgilbert

https://reviewboard.mozilla.org/r/48377/#review45921

Comment 67

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8744192 [details]
MozReview Request: Bug 1193526 - Add generated files. r=jgilbert

https://reviewboard.mozilla.org/r/48387/#review45923

Comment 68

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6423d55647ba
https://hg.mozilla.org/integration/mozilla-inbound/rev/ff47b156807b
https://hg.mozilla.org/integration/mozilla-inbound/rev/6ccf5bdf89f4
https://hg.mozilla.org/integration/mozilla-inbound/rev/df8471a4f59f
https://hg.mozilla.org/integration/mozilla-inbound/rev/730c8d4b4e38
https://hg.mozilla.org/integration/mozilla-inbound/rev/f73ccd27684a
https://hg.mozilla.org/integration/mozilla-inbound/rev/e510a704c1b0
https://hg.mozilla.org/integration/mozilla-inbound/rev/4688537106ce
https://hg.mozilla.org/integration/mozilla-inbound/rev/34836f8c1b9e
https://hg.mozilla.org/integration/mozilla-inbound/rev/9068b20e29a0
https://hg.mozilla.org/integration/mozilla-inbound/rev/ce229162e7b3

Comment 69

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/6423d55647ba
https://hg.mozilla.org/mozilla-central/rev/ff47b156807b
https://hg.mozilla.org/mozilla-central/rev/6ccf5bdf89f4
https://hg.mozilla.org/mozilla-central/rev/df8471a4f59f
https://hg.mozilla.org/mozilla-central/rev/730c8d4b4e38
https://hg.mozilla.org/mozilla-central/rev/f73ccd27684a
https://hg.mozilla.org/mozilla-central/rev/e510a704c1b0
https://hg.mozilla.org/mozilla-central/rev/4688537106ce
https://hg.mozilla.org/mozilla-central/rev/34836f8c1b9e
https://hg.mozilla.org/mozilla-central/rev/9068b20e29a0
https://hg.mozilla.org/mozilla-central/rev/ce229162e7b3

Comment 70

[Ryan VanderMeulen [:RyanVM]]

Was re-disabling the webgl conformance suite on e10s an intended change from this?

Comment 71

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #70)
> Was re-disabling the webgl conformance suite on e10s an intended change from
> this?

No, mochitest-errata is updated with correct setting. Maybe I forgot to re-generate the generted-mochitest.html with updated mochitest-errata.