1066467 - unstable MediaSource web platform tests

Status: RESOLVED WORKSFORME

(Core :: Audio/Video: Playback, defect, P5)

Description

[Karl Tomlinson (:karlt)]

Some MediaSource web platform tests run with inconsistent results when media.mediasource.enabled is set.
The suggested approach for unstable tests [1] is to disable the tests.

[1] https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/README.md#disabling-tests

I intend to leave this bug open after disabling to track re-enabling the tests.

Comment 1

[Karl Tomlinson (:karlt)]

Testing against revision 4d1793da0b96, the following tests or subtests produce unstable results:

mediasource-config-change-webm-av-audio-bitrate.html
  Parent may PASS or TIMEOUT 
  [Tests webm audio bitrate changes in multiplexed content.] may PASS, or FAIL

mediasource-config-change-webm-av-video-bitrate.html
  Parent may PASS or TIMEOUT 
  [Tests webm video bitrate changes in multiplexed content.] may PASS, or FAIL

mediasource-getvideoplaybackquality.html
  Parent may TIMEOUT or FAIL
  [Test HTMLVideoElement.getVideoPlaybackQuality() with MediaSource API] may
    FAIL or TIMEOUT

mediasource-play.html
  Parent may TIMEOUT or PASS
  [Test normal playback case with MediaSource API] may FAIL or TIMEOUT

mediasource-redundant-seek.html
  [Test redundant fully prebuffered seek] may TIMEOUT or FAIL

mediasource-seek-beyond-duration.html
  [Test seeking beyond media duration] may FAIL or TIMEOUT

Comment 2

[Karl Tomlinson (:karlt)]

Tested on revisions in the range 4d1793da0b96 to d462269d88a9,
mediasource-config-change-* tests were found to cause TIMEOUT in the
immediately following test.

The -config-change-mp4- tests all TIMEOUT themselves when run individually.

With only mediasource-config-change-webm-a-bitrate.html and
mediasource-config-change-webm-v-bitrate.html TIMEOUTs in the next test were
intermittent.

With other -config-change-webm- tests, TIMEOUTs in subsequent tests were
consistent.

Comment 3

[Karl Tomlinson (:karlt)]

Tested on revision d462269d88a9, running any of the following tests

  mediasource-buffered.html.ini
  mediasource-duration.html.ini
  mediasource-play-then-seek-back.html.ini
  mediasource-redundant-seek.html
  mediasource-seek-beyond-duration.html.ini
  mediasource-seek-during-pending-seek.html.ini
  mediasource-sourcebuffer-mode.html.ini

causes failure of old-tests/submission/Microsoft/sandbox/sandbox_002.htm,
which runs much later in the testsuite:

Allow autoplay for HTML5 Video inside iframe with sandbox attribute if sandbox='allow-scripts'.
-----------------------------------------------------------------------------------------------
Expected PASS, got FAIL
event is undefined(stack: callback/<@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.htm:22:17
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:798:20
callback@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.htm:20:13
)

Comment 4

[Karl Tomlinson (:karlt)]

Attachment: disable MediaSource tests that are unstable or interact badly with other tests

This is disabling 23 of 41 tests, but it means we can be running 18 of them.

Comment 5

[Karl Tomlinson (:karlt)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1dbd58c232fd

https://tbpl.mozilla.org/?tree=Try&rev=370a0779930a

Comment 6

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/1dbd58c232fd

Comment 7

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Set to P5 because this is a tracking bug.

Comment 8

[Karl Tomlinson (:karlt)]

(In reply to Karl Tomlinson (:karlt) from comment #2)
> The -config-change-mp4- tests all TIMEOUT themselves when run individually.

These are now stable, but failing.  Enabled in
https://hg.mozilla.org/integration/mozilla-inbound/rev/5d07a03af0b0

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/5d07a03af0b0

Comment 10

[Jean-Yves Avenard [:jya]]

Attachment: update test results for mac

mac and windows >= vista should provide the same results now.

Comment 11

[Jean-Yves Avenard [:jya]]

Attachment: update test results for mac

Update web reftest to reflect changes. Sounds like more could be re-enabled. Rebase

Comment 12

[Karl Tomlinson (:karlt)]

Comment on attachment 8545582 [details] [diff] [review]
update test results for mac

>       TIMEOUT
>-

Please leave the blank line at the end as autogenerated results will add it again when the tests are next pulled from upstream.

(Only subtest annotations get a blank line.  Parent annotations don't have trailing blank lines.)

Comment 13

[Jean-Yves Avenard [:jya]]

Attachment: update test results for mac

Carrying r+

Comment 14

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7c66101dc655

Comment 15

[Jean-Yves Avenard [:jya]]

Bug 1125776 should have made most of those tests now return reliable results.

Comment 16

[Jean-Yves Avenard [:jya]]

with bug 1171379, hopefully it won't be unstable anymore..

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:jya, maybe it's time to close this bug?

Comment 18

[Jean-Yves Avenard [:jya]]

the aim was to find the intermittents once the new MSE code went it. obviously worked