1241943 - Reenable webgl tests in web-platform-tests

Status: UNCONFIRMED

(Testing :: web-platform-tests, defect)

Description

[James Graham [:jgraham]]

A copy of the WebGL conformance suite v 1.0.3 has been imported into Web Platform Tests. Running this would have several advantages over the mochitest version we already have:

* It's newer
* It's easier to keep up with future changes (because the metadata about tests we pass is seperate from the actual tests themselves, and the upstream import seems to be entirely scripted)

However this is not currently enabled for a number of reasons:

* Long runtime caused chunks to timeout
* Full machine crashes on OSX 10.10.5
* Not clear that WebGL is actually correctly enabled for these tests (does mochitest-gl run under different hardware or another non-standard configuration).

These problems need to be solved in order to re-enable the tests.

Comment 1

[Kelsey Gilbert [:jgilbert]]

I do not think this is the right way forward.

> * It's newer
> * It's easier to keep up with future changes (because the metadata about tests we pass is seperate from the actual tests themselves, and the upstream import seems to be entirely scripted)

Bug 860999 solves both of these, without injecting additional friction into the downstreaming process.
We intend to continue to use the tests from Khronos directly, unless the repo moves. (which it is unlikely to do)

I'm marking this WONTFIX (maybe INVALID?) so we don't accidentally work on something we don't want.

Feel free to elaborate on why you think this bug may be worth reconsidering.

Comment 2

[James Graham [:jgraham]]

So it's not clear to me what the wider context here is; I think Ms2ger knows more about the addition of these tests to wpt. But afaict Bug 860999 doesn't actually solve the second problem because (unless I am missing something) we have to patch all the tests with annotations. The wpt harness is designed specifically to support tests that don't pass without having to change the tests themselves, so future upgrades become trivial (we import the new tests, update the metadata automatically from a try run, and that's it).

FWIW I really think Khronos should stop working in their own repo here. I don't see how it's providing any value, but it certainly causes problems because we have infrastructure to update and run web-platform-tests already.

Comment 3

[Kelsey Gilbert [:jgilbert]]

(In reply to James Graham [:jgraham] from comment #2)
> So it's not clear to me what the wider context here is; I think Ms2ger knows
> more about the addition of these tests to wpt. But afaict Bug 860999 doesn't
> actually solve the second problem because (unless I am missing something) we
> have to patch all the tests with annotations. The wpt harness is designed
> specifically to support tests that don't pass without having to change the
> tests themselves, so future upgrades become trivial (we import the new
> tests, update the metadata automatically from a try run, and that's it).

That's what the mochitest ini files do already, even without the new bug, which makes things even easier to upgrade. These failures are not portable, so we don't gain anything from standardizing with other vendors. (if indeed WPTs allow for this)

> FWIW I really think Khronos should stop working in their own repo here. I
> don't see how it's providing any value, but it certainly causes problems
> because we have infrastructure to update and run web-platform-tests already.

This is not the right forum for this discussion. It sounds like you may be missing background about WebGL. Let's have a chat so I can more quickly answer your questions in this vein. In short, it's a Khronos API, and as such the API accessories (tests) will remain with Khronos. Since Khronos continues to be the primary source for these things, we're going to continue to track the primary source.

Let's place this in the more neutral 'unconfirmed', since the need for this bug is not confirmed.

Comment 4

[Boris Zbarsky [:bzbarsky]]

> That's what the mochitest ini files do already

Here's a concrete example of where mochitest.ini files fall down.  For the crashing-because-they-touch-the-network tests (see bug 1333495) if they got fixed upstream we would just continue skipping them, because why wouldn't we?

On the other hand, in wpt they are annotated as a crash.  If they stop crashing that will come up as an unexpected pass (or a failure if they continue to fail in some way other than a crash), so if the tests are fixed upstream we will notice and reenable.