Closed Bug 996368 Opened 6 years ago Closed 6 years ago

Import W3C platform tests for MediaSource (based on Blink's tests)

Categories

(Core :: Audio/Video, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla35

People

(Reporter: kinetik, Assigned: karlt)

References

(Blocks 1 open bug, )

Details

Attachments

(1 file)

They'll need to be reviewed during import to make sure they agree with our reading of the spec (and either the test or spec clarified), and until we've implemented the complete spec I expect many of them will fail and need to be disabled until the missing functionality is implemented.
James,

These look like a subset of the web-platform tests that you are already working on importing. Can you comment here with how best to proceed with this?

The current web platform tests are being run on Cedar while they are stabilized before being rolled out to all trees. You can see the cedar jobs here: https://tbpl.mozilla.org/?tree=Cedar&jobname=web-
So, in the medium term the right solution is to run these along with all the other web-platform-tests. This import will be kept up to date so we will get new tests as soon as they reach the upstream repository, and we have machinery for dealing with tests that fail without having to disable them (basically we store the expected result for every test that doesn't pass).

I saw some media source tests in the update that I just pulled, so you should be able to see them running on cedar in the near future.

The code for web-platform-tests is in review and the plan is to roll it out to a few more trees, particularly try, once that is done. It will be enabled on m-c once we are confident it isn't going to cause more than its fair share of intermittent oranges.

If you need a short term solution, it is possible to import web-platform-tests on top of Mochitests, but that has a number of disadvantages, notably that it is relatively hard to keep up to date. It also lacks some advanced features, e.g. any kind of non-static response, that may or may not be needed by your tests. I'm not exactly sure how that import process works, but Ms2ger should know.

Also, I strongly encourage you to do the upstream review for PRs that you want to land. Finding expert reviewers is probably the biggest bottleneck to getting tests landed in web-platform-tests.
Depends on: 945222
I'd been hoping to avoid updating my copy; let me know if this is urgent, though, and the referenced PR has been reviewed and landed upstream.
Depends on: 1066467
https://github.com/w3c/web-platform-tests/pull/1238 is a PR for importing more Blink tests into wpt. It would be great if someone who knows the spec could review.
I'm happy to look at that PR at some stage, but higher priority for me right now is running the tests we already have with MediaSource enabled.  Once I sort out the issues there, I'll likely have better understanding of the harness so as to review new tests.
Assignee: nobody → karlt
OK. Is running then with that enabled something that you need help with on the automation side, or is it just about making code changes to ensure that the tests are stable?
You've been very helpful, thanks James.

I have a subset of tests running with stable results, but I've had to disable a large number of them because of an interaction with old-tests/submission/Microsoft/sandbox/sandbox_002.htm:

Allow autoplay for HTML5 Video inside iframe with sandbox attribute if sandbox='allow-scri
pts'.
------------------------------------------------------------------------------------------
-----
Expected PASS, got FAIL
event is undefined(stack: callback/<@http://web-platform.test:8000/old-tests/submission/Mi
crosoft/sandbox/sandbox_002.htm:22:17
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:798:20
callback@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.
htm:20:13
)

This is more than likely a bug in Gecko, which I'd like to investigate.

Is there a way to get more output from the tests?
i.e. can I get the harness to log every assert_* both when it passes and when it fails?
Depends on: 1069009
Depends on: 1069113
This includes changes to web platform test expected results.

Removal of subtests annotations in meta .ini files means those subtests are
passing.  Removal of a whole file means the whole test is passing.
Attachment #8491345 - Flags: review?(kinetik)
Comment on attachment 8491345 [details] [diff] [review]
Enable Media Source Extensions for testing

Review of attachment 8491345 [details] [diff] [review]:
-----------------------------------------------------------------

FWIW the changes look like they match the expected syntax. I obviously don't know if the results are correct given the current state of the implemntation.
Attachment #8491345 - Flags: feedback+
Attachment #8491345 - Flags: review?(kinetik) → review+
https://hg.mozilla.org/mozilla-central/rev/113ecae356de
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
Depends on: 1098200
Flags: qe-verify-
You need to log in before you can comment on or make changes to this bug.