1052240 - Generate a separate mochitest for each WebGL Conformance Test test file

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect)

Description

[Kelsey Gilbert [:jgilbert]]

This is part of the migration to using as much non-webgl-specific stuff as possible, in regards to how we run mochitests.

Having these tests separate allows us to try to leverage existing mochitest manifest files as much as possible.

Hopefully this helps with some of our intermittent failures, as well.

Comment 1

[Kelsey Gilbert [:jgilbert]]

Attachment: patch 1: Add generator and template files

Comment 2

[Kelsey Gilbert [:jgilbert]]

Attachment: patch 2: Generated files

Just review that it seems to be generating the right files. No need to review each file!

Comment 3

[Kelsey Gilbert [:jgilbert]]

Attachment: patch 3: Fix test that was failing locally.

This might actually fix an intermittent orange somewhere.

Comment 4

[Kelsey Gilbert [:jgilbert]]

Try run:
https://tbpl.mozilla.org/?tree=Try&rev=be89807b800e

This should have a bunch of failures, since I haven't included the logic for skipped/failing-tests-per-driver/os-config, yet.

Comment 5

[u480271]

Comment on attachment 8471308 [details] [diff] [review]
patch 2: Generated files

Review of attachment 8471308 [details] [diff] [review]:
-----------------------------------------------------------------

Am I supposed to review all of these?

Comment 6

[u480271]

Comment on attachment 8471308 [details] [diff] [review]
patch 2: Generated files

Review of attachment 8471308 [details] [diff] [review]:
-----------------------------------------------------------------

I visually inspected a sample of files. They look like they match the templates.

Comment 7

[Kelsey Gilbert [:jgilbert]]

'fail-if' (silently!) doesn't actually do anything in mochitest.ini files, even though the manifest parser picks up on it properly. This is bug 987849.

We laugh so that we do not cry.

Comment 8

[Kelsey Gilbert [:jgilbert]]

https://tbpl.mozilla.org/?tree=Try&rev=27d90bdc635b

* Linux x64 Debug is randomly crashing about 15% of the time, seemingly consistently in the texture tests.
* Android 2.3 Opt is chain-bluing as the device goes unresponsive. (hardcrash?)
* B2G ICS Emulator Opt appears to be consistently crashing.

Comment 9

[Kelsey Gilbert [:jgilbert]]

The last test printed before the ICS Emu runs crash is 'conformance/misc/type-conversion-test.html'.

Comment 10

[Kelsey Gilbert [:jgilbert]]

Android 2.3 failures also have last-printed-tests saying 'type-conversion-test'. So, uh, I guess we disable that for now. :p

Comment 11

[Kelsey Gilbert [:jgilbert]]

(In reply to Jeff Gilbert [:jgilbert] from comment #10)
> Android 2.3 failures also have last-printed-tests saying
> 'type-conversion-test'. So, uh, I guess we disable that for now. :p

Additionally, it looks like misc/object-deletion-behavior also UNEXPECTED-FAILs.

Comment 12

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

This is great -- out of curiosity, is there much of a time difference between running them as individual mochitests vs running them all in one go?

Comment 13

[Kelsey Gilbert [:jgilbert]]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #12)
> This is great -- out of curiosity, is there much of a time difference
> between running them as individual mochitests vs running them all in one go?

I'll gather some more data, but it looks like it's about the same.
Related, the average Android 2.3 Opt test takes 6-7s to run. :|

Comment 14

[Kelsey Gilbert [:jgilbert]]

Attachment: patch 4: Rearrange some code

Comment 15

[Kelsey Gilbert [:jgilbert]]

Attachment: patch 5: Mark tests to skip/fail, and re-gen manifest.

Comment 16

[Kelsey Gilbert [:jgilbert]]

Try is finally clear:
https://tbpl.mozilla.org/?tree=Try&rev=d92dae9c6f59

I retriggered a bunch of tests to try to assure I've gotten the worst of the intermittent failures.

Comment 17

[Kelsey Gilbert [:jgilbert]]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/a209d60bed50
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/3708cca072dc
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/b2679f354c15
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/429de5ae1110
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/b695c5227ce6

Comment 18

[Nigel Babu [:nigelb]]

Hi, I think you've caused B2G ICS Emulator Debug M7 bustage. See https://tbpl.mozilla.org/php/getParsedLog.php?id=46532075&tree=Mozilla-Inbound

Comment 19

[Nigel Babu [:nigelb]]

Backed out

https://hg.mozilla.org/integration/mozilla-inbound/rev/fd3f7d35e751
https://hg.mozilla.org/integration/mozilla-inbound/rev/5352be2b9469
https://hg.mozilla.org/integration/mozilla-inbound/rev/3d5e163559c0
https://hg.mozilla.org/integration/mozilla-inbound/rev/6e7a6059a48d
https://hg.mozilla.org/integration/mozilla-inbound/rev/a59c073a342a

Comment 20

[Ed Morley [:emorley]]

(In reply to Jeff Gilbert [:jgilbert] from comment #0)
> Hopefully this helps with some of our intermittent failures, as well.

This bug will definitely help with the bug suggestions on TBPL/treeherder too, so thank you :-)

Comment 21

[Ed Morley [:emorley]]

Also frequent intermittent:
https://tbpl.mozilla.org/php/getParsedLog.php?id=46537126&tree=Mozilla-Inbound

Comment 22

[Kelsey Gilbert [:jgilbert]]

(In reply to Ed Morley [:edmorley] from comment #21)
> Also frequent intermittent:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=46537126&tree=Mozilla-
> Inbound

I don't believe this is my fault per se. This adds about 360 test files, so maybe it's bloating the log, and things need to be split more.

Comment 23

[Kelsey Gilbert [:jgilbert]]

Attachment: patch 5: Mark tests to skip/fail, and re-gen manifest.

More failing tests.

Comment 24

[Kelsey Gilbert [:jgilbert]]

This should do it:
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/f7c5431f1c82
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/ce97309bb8f5
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/12b979c3b7a7
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/99302ed2d6ad
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/2dd8e4e95c55

If more tests crash, they should be marked for skipping.
To do this, just add an entry to mochitest-errata.ini, and run generate-wrapper-and-manifest.py.
(Or just add it to _mochitest.ini directly, and needinfo me and I'll fix it properly.)

Comment 25

[Kartikaya Gupta (email:[e-mail])]

It looks like test_conformance__textures__tex-image-and-sub-image-2d-with-image.html is failing pretty consistently on B2G ICS Emulator Debug. Here's a sample log:

https://tbpl.mozilla.org/php/getParsedLog.php?id=46610290&tree=Mozilla-Inbound

I'll land a patch to _mochitest.ini to disable that one on b2g.

Comment 26

[Kartikaya Gupta (email:[e-mail])]

test_conformance__textures__tex-image-and-sub-image-2d-with-video.html also has failed a couple of times on windows.

https://tbpl.mozilla.org/php/getParsedLog.php?id=46634504&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=46634463&tree=Mozilla-Inbound

It seems very intermittent at this point so I'll leave it enabled but it's something you should look into.

Comment 27

[Kartikaya Gupta (email:[e-mail])]

Here's the patch for comment 25.

https://hg.mozilla.org/integration/mozilla-inbound/rev/fbd0bffddd86

Comment 28

[Kartikaya Gupta (email:[e-mail])]

Also looks like test_conformance__textures__tex-image-and-sub-image-2d-with-image-data.html is still failing on B2G ICS Emulator Debug, in much the same way that test_conformance__textures__tex-image-and-sub-image-2d-with-image.html is failing.

Comment 29

[Nigel Babu [:nigelb]]

kats, I think the bustage fix caused a bustage. m8 on b2g was greeen for 2 pushes after https://hg.mozilla.org/integration/mozilla-inbound/rev/1135f5842f1f and then busted after fbd0bffddd86. I'm suspecting this push is the cause. I've triggered a bunch of retries. I might back it out later to see if it helps.

Comment 30

[Kartikaya Gupta (email:[e-mail])]

(In reply to Nigel Babu [:nigelb] from comment #29)
> kats, I think the bustage fix caused a bustage. m8 on b2g was greeen for 2
> pushes after
> https://hg.mozilla.org/integration/mozilla-inbound/rev/1135f5842f1f and then
> busted after fbd0bffddd86.

I think my patch didn't actually fix much because instead of crashing on one test it now crashes on the next one. I don't think my patch made things worse though; the two green pushes were likely just intermittent passes. You can retrigger those if you like, I bet they go orange on retriggers.

Comment 31

[Nigel Babu [:nigelb]]

Oh joy. Indeed. I see random greens on some retries, but mostly orange.

Comment 32

[Kelsey Gilbert [:jgilbert]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #30)
> (In reply to Nigel Babu [:nigelb] from comment #29)
> > kats, I think the bustage fix caused a bustage. m8 on b2g was greeen for 2
> > pushes after
> > https://hg.mozilla.org/integration/mozilla-inbound/rev/1135f5842f1f and then
> > busted after fbd0bffddd86.
> 
> I think my patch didn't actually fix much because instead of crashing on one
> test it now crashes on the next one. I don't think my patch made things
> worse though; the two green pushes were likely just intermittent passes. You
> can retrigger those if you like, I bet they go orange on retriggers.

Yeah, crashes can do this. We just need to keep skipping tests until it passes, for now.

Comment 33

[Carsten Book [:Tomcat]]

hey guys,

had to back out this 5 patch from jeff and also the patch from patch, since we got so much frequent test failures

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/0adf0a108a7f
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/f0592de4261c
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/c22358426fd3
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/6074e6792a95
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/5e51b5ee39bf
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/1fcffcc9fc4a

Comment 34

[Ryan VanderMeulen [:RyanVM]]

Looks like Windows was hitting intermittent crashes as well:
https://tbpl.mozilla.org/php/getParsedLog.php?id=46612374&tree=Mozilla-Inbound

Comment 35

[Kelsey Gilbert [:jgilbert]]

(In reply to Carsten Book [:Tomcat] from comment #33)
> hey guys,
> 
> had to back out this 5 patch from jeff and also the patch from patch, since
> we got so much frequent test failures
> 
> remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/0adf0a108a7f
> remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/f0592de4261c
> remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/c22358426fd3
> remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/6074e6792a95
> remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/5e51b5ee39bf
> remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/1fcffcc9fc4a

Please give me a list of the failures. This looked quite green on Try.

Comment 36

[Carsten Book [:Tomcat]]

here are 2 as example

https://tbpl.mozilla.org/php/getParsedLog.php?id=46678964&tree=Mozilla-Inbound
and
https://tbpl.mozilla.org/php/getParsedLog.php?id=46676917&tree=Mozilla-Inbound

Comment 37

[Kelsey Gilbert [:jgilbert]]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/e156a5c3f783
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/0b576e3488f3
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/745d2a713b74
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/121a0dce1fc3
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/aeec83c02423
I disabled B2G for now.

Comment 38

[Kelsey Gilbert [:jgilbert]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #34)
> Looks like Windows was hitting intermittent crashes as well:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=46612374&tree=Mozilla-
> Inbound

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/2f858cc61182

Comment 39

[Kelsey Gilbert [:jgilbert]]

Please DON'T back this out for intermittents. PLEASE mark them for skipping in _mochitest.ini.

Without becoming an even worse Try abuser[1], I can't reasonably identify intermittent tests anymore.
Just mark them skipped like any new intermittent test failure.

[1] During the week when I was seriously testing this, I was Try High Scores #4. :|

Comment 40

[Kelsey Gilbert [:jgilbert]]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/1f81a6902e96

Comment 41

[Kelsey Gilbert [:jgilbert]]

One OSX 10.6 M2 run had a failure:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=2f858cc61182

I retriggered 10 each of 10.6 and 10.8 opt, and no other failures. I retriggered 10 more each, to see if it's closer to a 5% fail rate.

Comment 42

[Kelsey Gilbert [:jgilbert]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=47081252&tree=Mozilla-Inbound#error1
https://tbpl.mozilla.org/?rev=2b60f30db362&tree=Mozilla-Inbound

  https://hg.mozilla.org/integration/mozilla-inbound/rev/e9a97ae0e909

Comment 43

[Kelsey Gilbert [:jgilbert]]

Attachment: patch 10: Fix the generator for use on Windows

Comment 44

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/e156a5c3f783
https://hg.mozilla.org/mozilla-central/rev/0b576e3488f3
https://hg.mozilla.org/mozilla-central/rev/745d2a713b74
https://hg.mozilla.org/mozilla-central/rev/121a0dce1fc3
https://hg.mozilla.org/mozilla-central/rev/aeec83c02423
https://hg.mozilla.org/mozilla-central/rev/2f858cc61182
https://hg.mozilla.org/mozilla-central/rev/1f81a6902e96
https://hg.mozilla.org/mozilla-central/rev/e9a97ae0e909

Comment 45

[Kelsey Gilbert [:jgilbert]]

A bunch of retriggers after skipping the win8 texture-npot intermittent:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=e9a97ae0e909

Comment 46

[Kelsey Gilbert [:jgilbert]]

OSX canvas__drawingbuffer-static-canvas-test.html had about 15% crash rate.
Win8 textures__texture-npot-video.html had about 20% crash rate.

Comment 47

[Kelsey Gilbert [:jgilbert]]

Win8, Linux 32 Opt now appears to have a failure rate of >5% (20/20 green).
Android 4.0 Opt has 1/18 failures win webgl atm, I retriggered enough for 30 runs. (This is the blue)
I retriggered enough for 20 runs of OSX 10.8 Opt, to check it for common failures.

Comment 48

[Kelsey Gilbert [:jgilbert]]

On Android 4.0 Opt, blue device-unresponsive in webgl:
_wrappers/test_conformance__more__functions__drawElements.html (~1/30)
_wrappers/test_conformance__more__conformance__quickCheckAPI-L_S.html (~1/30)

Retriggering more.

Comment 49

[Kelsey Gilbert [:jgilbert]]

OSX 10.8 Opt looks clean after 20 runs.

Comment 50

[Kelsey Gilbert [:jgilbert]]

 https://hg.mozilla.org/integration/mozilla-inbound/rev/c4cb0bd2dbf1

Comment 51

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c4cb0bd2dbf1