Some MediaSource web platform tests run with inconsistent results when media.mediasource.enabled is set. The suggested approach for unstable tests  is to disable the tests.  https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/README.md#disabling-tests I intend to leave this bug open after disabling to track re-enabling the tests.
Testing against revision 4d1793da0b96, the following tests or subtests produce unstable results: mediasource-config-change-webm-av-audio-bitrate.html Parent may PASS or TIMEOUT [Tests webm audio bitrate changes in multiplexed content.] may PASS, or FAIL mediasource-config-change-webm-av-video-bitrate.html Parent may PASS or TIMEOUT [Tests webm video bitrate changes in multiplexed content.] may PASS, or FAIL mediasource-getvideoplaybackquality.html Parent may TIMEOUT or FAIL [Test HTMLVideoElement.getVideoPlaybackQuality() with MediaSource API] may FAIL or TIMEOUT mediasource-play.html Parent may TIMEOUT or PASS [Test normal playback case with MediaSource API] may FAIL or TIMEOUT mediasource-redundant-seek.html [Test redundant fully prebuffered seek] may TIMEOUT or FAIL mediasource-seek-beyond-duration.html [Test seeking beyond media duration] may FAIL or TIMEOUT
Tested on revisions in the range 4d1793da0b96 to d462269d88a9, mediasource-config-change-* tests were found to cause TIMEOUT in the immediately following test. The -config-change-mp4- tests all TIMEOUT themselves when run individually. With only mediasource-config-change-webm-a-bitrate.html and mediasource-config-change-webm-v-bitrate.html TIMEOUTs in the next test were intermittent. With other -config-change-webm- tests, TIMEOUTs in subsequent tests were consistent.
Tested on revision d462269d88a9, running any of the following tests mediasource-buffered.html.ini mediasource-duration.html.ini mediasource-play-then-seek-back.html.ini mediasource-redundant-seek.html mediasource-seek-beyond-duration.html.ini mediasource-seek-during-pending-seek.html.ini mediasource-sourcebuffer-mode.html.ini causes failure of old-tests/submission/Microsoft/sandbox/sandbox_002.htm, which runs much later in the testsuite: Allow autoplay for HTML5 Video inside iframe with sandbox attribute if sandbox='allow-scripts'. ----------------------------------------------------------------------------------------------- Expected PASS, got FAIL event is undefined(stack: callback/<@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.htm:22:17 Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:798:20 callback@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.htm:20:13 )
Created attachment 8491336 [details] [diff] [review] disable MediaSource tests that are unstable or interact badly with other tests This is disabling 23 of 41 tests, but it means we can be running 18 of them.
Set to P5 because this is a tracking bug.
(In reply to Karl Tomlinson (:karlt) from comment #2) > The -config-change-mp4- tests all TIMEOUT themselves when run individually. These are now stable, but failing. Enabled in https://hg.mozilla.org/integration/mozilla-inbound/rev/5d07a03af0b0
Created attachment 8545554 [details] [diff] [review] update test results for mac mac and windows >= vista should provide the same results now.
Created attachment 8545582 [details] [diff] [review] update test results for mac Update web reftest to reflect changes. Sounds like more could be re-enabled. Rebase
Comment on attachment 8545582 [details] [diff] [review] update test results for mac > TIMEOUT >- Please leave the blank line at the end as autogenerated results will add it again when the tests are next pulled from upstream. (Only subtest annotations get a blank line. Parent annotations don't have trailing blank lines.)
Bug 1125776 should have made most of those tests now return reliable results.