unstable MediaSource web platform tests

ASSIGNED
Assigned to

Status

()

Core
Audio/Video: Playback
P5
normal
ASSIGNED
3 years ago
2 years ago

People

(Reporter: karlt, Assigned: jya)

Tracking

(Depends on: 2 bugs, Blocks: 1 bug, {intermittent-failure, leave-open})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [disabled])

Attachments

(2 attachments, 2 obsolete attachments)

(Reporter)

Description

3 years ago
Some MediaSource web platform tests run with inconsistent results when media.mediasource.enabled is set.
The suggested approach for unstable tests [1] is to disable the tests.

[1] https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/README.md#disabling-tests

I intend to leave this bug open after disabling to track re-enabling the tests.
(Reporter)

Comment 1

3 years ago
Testing against revision 4d1793da0b96, the following tests or subtests produce unstable results:

mediasource-config-change-webm-av-audio-bitrate.html
  Parent may PASS or TIMEOUT 
  [Tests webm audio bitrate changes in multiplexed content.] may PASS, or FAIL

mediasource-config-change-webm-av-video-bitrate.html
  Parent may PASS or TIMEOUT 
  [Tests webm video bitrate changes in multiplexed content.] may PASS, or FAIL

mediasource-getvideoplaybackquality.html
  Parent may TIMEOUT or FAIL
  [Test HTMLVideoElement.getVideoPlaybackQuality() with MediaSource API] may
    FAIL or TIMEOUT

mediasource-play.html
  Parent may TIMEOUT or PASS
  [Test normal playback case with MediaSource API] may FAIL or TIMEOUT

mediasource-redundant-seek.html
  [Test redundant fully prebuffered seek] may TIMEOUT or FAIL

mediasource-seek-beyond-duration.html
  [Test seeking beyond media duration] may FAIL or TIMEOUT
(Reporter)

Comment 2

3 years ago
Tested on revisions in the range 4d1793da0b96 to d462269d88a9,
mediasource-config-change-* tests were found to cause TIMEOUT in the
immediately following test.

The -config-change-mp4- tests all TIMEOUT themselves when run individually.

With only mediasource-config-change-webm-a-bitrate.html and
mediasource-config-change-webm-v-bitrate.html TIMEOUTs in the next test were
intermittent.

With other -config-change-webm- tests, TIMEOUTs in subsequent tests were
consistent.
(Reporter)

Comment 3

3 years ago
Tested on revision d462269d88a9, running any of the following tests

  mediasource-buffered.html.ini
  mediasource-duration.html.ini
  mediasource-play-then-seek-back.html.ini
  mediasource-redundant-seek.html
  mediasource-seek-beyond-duration.html.ini
  mediasource-seek-during-pending-seek.html.ini
  mediasource-sourcebuffer-mode.html.ini

causes failure of old-tests/submission/Microsoft/sandbox/sandbox_002.htm,
which runs much later in the testsuite:

Allow autoplay for HTML5 Video inside iframe with sandbox attribute if sandbox='allow-scripts'.
-----------------------------------------------------------------------------------------------
Expected PASS, got FAIL
event is undefined(stack: callback/<@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.htm:22:17
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:798:20
callback@http://web-platform.test:8000/old-tests/submission/Microsoft/sandbox/sandbox_002.htm:20:13
)
(Reporter)

Comment 4

3 years ago
Created attachment 8491336 [details] [diff] [review]
disable MediaSource tests that are unstable or interact badly with other tests

This is disabling 23 of 41 tests, but it means we can be running 18 of them.
Attachment #8491336 - Flags: review?(kinetik)
Attachment #8491336 - Flags: review?(kinetik) → review+
(Reporter)

Comment 5

3 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/1dbd58c232fd

https://tbpl.mozilla.org/?tree=Try&rev=370a0779930a
https://hg.mozilla.org/mozilla-central/rev/1dbd58c232fd
(Reporter)

Updated

3 years ago
Blocks: 778617
(Reporter)

Updated

3 years ago
Depends on: 1080958
(Reporter)

Updated

3 years ago
Depends on: 1082206
(Reporter)

Updated

3 years ago
Depends on: 1085247
(Assignee)

Updated

3 years ago
See Also: → bug 1093945
Set to P5 because this is a tracking bug.
Priority: -- → P5
(Reporter)

Comment 8

3 years ago
(In reply to Karl Tomlinson (:karlt) from comment #2)
> The -config-change-mp4- tests all TIMEOUT themselves when run individually.

These are now stable, but failing.  Enabled in
https://hg.mozilla.org/integration/mozilla-inbound/rev/5d07a03af0b0
https://hg.mozilla.org/mozilla-central/rev/5d07a03af0b0
(Assignee)

Comment 10

3 years ago
Created attachment 8545554 [details] [diff] [review]
update test results for mac

mac and windows >= vista should provide the same results now.
Attachment #8545554 - Flags: review?(karlt)
(Assignee)

Updated

3 years ago
Assignee: karlt → jyavenard
Status: NEW → ASSIGNED
(Assignee)

Comment 11

3 years ago
Created attachment 8545582 [details] [diff] [review]
update test results for mac

Update web reftest to reflect changes. Sounds like more could be re-enabled. Rebase
Attachment #8545582 - Flags: review?(karlt)
(Assignee)

Updated

3 years ago
Attachment #8545554 - Attachment is obsolete: true
Attachment #8545554 - Flags: review?(karlt)
(Assignee)

Updated

3 years ago
Assignee: jyavenard → karlt
(Reporter)

Comment 12

3 years ago
Comment on attachment 8545582 [details] [diff] [review]
update test results for mac

>       TIMEOUT
>-

Please leave the blank line at the end as autogenerated results will add it again when the tests are next pulled from upstream.

(Only subtest annotations get a blank line.  Parent annotations don't have trailing blank lines.)
Attachment #8545582 - Flags: review?(karlt) → review+
(Assignee)

Comment 13

3 years ago
Created attachment 8546298 [details] [diff] [review]
update test results for mac

Carrying r+
(Assignee)

Updated

3 years ago
Attachment #8545582 - Attachment is obsolete: true
(Assignee)

Updated

3 years ago
Assignee: karlt → jyavenard
https://hg.mozilla.org/mozilla-central/rev/7c66101dc655
(Reporter)

Updated

3 years ago
Depends on: 1128069
(Assignee)

Comment 15

3 years ago
Bug 1125776 should have made most of those tests now return reliable results.

Updated

3 years ago
Depends on: 1134888
(Reporter)

Updated

3 years ago
Depends on: 1143650
(Reporter)

Updated

3 years ago
Depends on: 1143999
(Reporter)

Updated

3 years ago
Depends on: 1154881
(Reporter)

Updated

3 years ago
Depends on: 1154016
(Assignee)

Comment 16

2 years ago
with bug 1171379, hopefully it won't be unstable anymore..
Depends on: 1171379
Component: Audio/Video → Audio/Video: Playback
You need to log in before you can comment on or make changes to this bug.