984523 - Android 4.0 Opt mochitest-gl is skipping all of the content/canvas/test/webgl-conformance tests

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Geoff Brown [:gbrown]]

Since rev 91f4d6270e24 landed from bug 970925, Android 4.0 Opt mochitest-gl has been running fewer tests than it did in the past.

Consider:

https://tbpl.mozilla.org/php/getParsedLog.php?id=35989329&tree=Mozilla-Inbound&full=1
03:13:02     INFO -  63 INFO Passed:  7355
03:13:02     INFO -  64 INFO Failed:  0
03:13:02     INFO -  65 INFO Todo:    54
03:13:02     INFO -  66 INFO Slowest: 4109ms - /tests/content/canvas/test/webgl/non-conf-tests/test_depth_readpixels.html
03:13:02     INFO -  67 INFO SimpleTest FINISHED
03:13:02     INFO -  INFO | automation.py | Application ran for: 0:04:42.791477

vs.

https://tbpl.mozilla.org/php/getParsedLog.php?id=35994264&tree=Mozilla-Inbound&full=1
05:24:06     INFO -  33 INFO Passed:  24
05:24:06     INFO -  34 INFO Failed:  0
05:24:06     INFO -  35 INFO Todo:    1
05:24:06     INFO -  36 INFO Slowest: 2988ms - /tests/content/canvas/test/webgl/non-conf-tests/test_depth_readpixels.html
05:24:06     INFO -  37 INFO SimpleTest FINISHED
05:24:06     INFO -  INFO | automation.py | Application ran for: 0:00:22.194483

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I suspect this is because we do build time filtering of the test files, this means that we don't copy as many files to the tests.zip and we don't have as many to run.  The problem is by using android.json we could skip everything, but now that we are not using it these tests start to run on android mochitest and fail.

Either we put an entry for the webgl directory in android.json, or wait a couple days until we are filtering at the test harness level and not at the build level.

Comment 2

[Kelsey Gilbert [:jgilbert]]

24 tests sounds like it's just running webgl/non-conf-tests, and none of webgl/conformance.

(In reply to Joel Maher (:jmaher) from comment #1)
> I suspect this is because we do build time filtering of the test files, this
> means that we don't copy as many files to the tests.zip and we don't have as
> many to run.  The problem is by using android.json we could skip everything,
> but now that we are not using it these tests start to run on android
> mochitest and fail.
> 
> Either we put an entry for the webgl directory in android.json, or wait a
> couple days until we are filtering at the test harness level and not at the
> build level.

I'm willing to take this hit, but I'd put our tolerance limit at a week. Is there any chance this could be missed?

Is there documentation regarding these changes? Bug 981936 (if it sticks) will change the paths involved for both conformance and non-conformance WebGL tests.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

ok, the manifests changes have been landed, now we need to figure out how to turn this on and make it work for gl only.  possibly we could use this new idea of a subsuite (still needs to be implemented).

:ted, do you think a subsuite here for the gl tests is a good idea?

Comment 4

[Geoff Brown [:gbrown]]

It seems we now run tests in content/canvas/test/webgl-mochitest on Android 2.2/2.3/4.0, as part of the normal mochitest chunks. We run these same tests again on Android 2.3 and Android 4.0 in mochitest-gl. As far as I can tell, the content/canvas/test/webgl-conformance tests are not running for any Android.

Comment 5

[(not currently active) Ted Mielczarek]

We should be running from the test harness now, so is this fixed?

Comment 6

[Geoff Brown [:gbrown]]

This is not fixed.

Consider Android 4.0 Opt, M2: https://tbpl.mozilla.org/php/getParsedLog.php?id=36899560&full=1&branch=mozilla-inbound

14:07:01     INFO -  554 INFO TEST-START | /tests/content/canvas/test/webgl-mochitest/test_depth_readpixels.html

and Android 4.0 Opt, M-gl: https://tbpl.mozilla.org/php/getParsedLog.php?id=36899317&tree=Mozilla-Inbound&full=1

14:04:44     INFO -  1 INFO TEST-START | /tests/content/canvas/test/webgl-mochitest/test_depth_readpixels.html

but neither one runs any tests from http://hg.mozilla.org/mozilla-central/file/5dea2146a099/content/canvas/test/webgl-conformance

Comment 7

[(not currently active) Ted Mielczarek]

Okay. I guess we'll just use jmaher's subsuite work from bug 984930 then. I don't have any other great ideas for how to fix this.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

subsuite won't work as we have filtering at build time.  I am tempted to put android.json back to work until we are not filtering at build time and can solve this with subsuites.

Comment 9

[(not currently active) Ted Mielczarek]

Where are we still filtering at build time? I thought bug 938019 turned all that off?

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I was debugging stuff locally and my 'skip-if = debug' for browser-chrome stuff in devtools yielded filtering at build time :(  the tests.zip file didn't have any of the tests I wanted to debug further.

Comment 11

[(not currently active) Ted Mielczarek]

Huh, I thought I remembered that bug fixing that, but clearly it didn't. This should be a 1-liner:
http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/frontend/emitter.py#482

Comment 12

[Geoff Brown [:gbrown]]

Attachment: use android.json for excluding webgl; fix gl.json

> subsuite won't work as we have filtering at build time.  I am tempted to put 
> android.json back to work until we are not filtering at build time and can 
> solve this with subsuites.

Here's a patch to do just that.

I know that Armen is working to unblock the subsuite approach; if you think a subsuite fix is imminent, feel free to reject this approach.

This patch restores the webgl-conformance tests to the mochitest-gl job on Android 4.0, Android x86, and Android 2.3, and runs the webgl-mochitest tests exactly once on all Androids:

https://tbpl.mozilla.org/?tree=Try&rev=2e231369d503
https://tbpl.mozilla.org/?tree=Ash&rev=947a613d0ef4

There is a new crash on Android 4.0 m-gl, which I will report shortly; we'll need to address that before landing this.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8419038 [details] [diff] [review]
use android.json for excluding webgl; fix gl.json

Review of attachment 8419038 [details] [diff] [review]:
-----------------------------------------------------------------

lets do this for now, once we have all tests in tests.zip we can work on cleaning this up

Comment 14

[Geoff Brown [:gbrown]]

Attachment: Android 2.3 mozharness change to use gl.json for mochitest-gl

A minor change to the Android 2.3 mozharness config for mochitest-gl: use gl.json, like the pandas do. I verified this on ash.

Comment 15

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/61747f0ab82e

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/61747f0ab82e

Comment 17

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/mozharness/rev/99c4d4c961bd

Still need to leave open for the subsuite work.

Comment 18

[Armen [:armenzg]]

Merged into production and live.

Comment 19

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #17)
> https://hg.mozilla.org/build/mozharness/rev/99c4d4c961bd
> 
> Still need to leave open for the subsuite work.

That was sorted out long ago - all done now.