1298733 - Autophone - add support for WebGL tests

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: bug-1298733-v1.patch

Example run with local devices shows Interesting crashes.

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=aef078d44124e895091e7341ae348fe3bb0c2255&filter-searchStr=autophone&group_state=expanded

I think Mwglm is short enough that we can support it on production with our current inventory. Mwglc takes too long to support by default but we can support it on try.

Thoughts on names/symbols etc?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8785770 [details] [diff] [review]
bug-1298733-v1.patch

Review of attachment 8785770 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/mochitests-webgl-conf-settings.ini
@@ +25,5 @@
> +total_chunks = 10
> +
> +[preferences]
> +gfx.canvas.azure.backends=skia
> +gfx.canvas.azure.accelerated=true

I will let snorp comment on these prefs if they are proper, or if we have all of them.

@@ +29,5 @@
> +gfx.canvas.azure.accelerated=true
> +
> +[treeherder]
> +job_name = Autophone Mochitest WebGL conf
> +job_symbol = Mwglc

we assume WebGL == gl, on the tree, so could we just make this gl-c ?

::: configs/mochitests-webgl-mochitest-settings.ini
@@ +29,5 @@
> +gfx.canvas.azure.accelerated=true
> +
> +[treeherder]
> +job_name = Autophone Mochitest WebGL mochitest
> +job_symbol = Mwglm

I would like 'gl' as the symbol name

Comment 3

[Bob Clary [:bc] (inactive)]

The prefs came from the Skia tests and may not be appropriate. snorp will know.

The webgl-conf in treeherder appear as gl<chunk> but I think their actual symbols are Mgl<chunk>. They appear as gl<chunk> due to the way the Mochitests are combined in the output. We could make them gl<chunk> and they would appear similarly to the current webgl-conf tests. I don't actually see where the webgl-mochitest tests are being run any where. They should be different. Perhaps just glm.

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

I like 'gl' as the name too.

The crashes we'll need to get fixed up, but I'm not sure about the other failures. It could be that those features are just not present on the hardware and we don't check for that? Input from Jeff here would be helpful.

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: bug-1298733-v2.patch

-+job_symbol = Mwglc
++job_symbol = gl

-+job_symbol = Mwglm
++job_symbol = glm

A previous patch in bug 1298732 for using Treeherder to parse the logs didn't handle the marking of failing tests properly. A new local run with the new job symbols is in progress which better marks failing tests. The results are available at https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=506facea63169a29e04eb140663da1730052db64&filter-searchStr=autophone&group_state=expanded&selectedJob=4409398

See bug 1298732 for more details on the log parsing issues.

Carrying forward the r+

Comment 6

[Kelsey Gilbert [:jgilbert]]

If it's different hardware, it'll have different failures. (and potentially different crashes)

Comment 7

[Bob Clary [:bc] (inactive)]

Attachment: samsung gs3/nexus 6p webgl-crashes.txt

Up to date local test run:

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=b7f7ae14590aced450bb0b0469dfb38edd2c0ace&filter-searchStr=autophone&group_state=expanded

Comment 8

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/857109cc1056ae9b948df3c96f9585eb1438431a
https://github.com/mozilla/autophone/commit/62d478686ca14a02e8e19c5134a809e93f613fd3

Comment 9

[Bob Clary [:bc] (inactive)]

deployed 2016-09-04 ~13:40 PDT

leave open until we run in production.

Comment 10

[Bob Clary [:bc] (inactive)]

These are available on try and won't be available for CI until they are green and/or we have more devices for testing. Marking fixed.