1764293 - Embedded videos on twitter don't play

Status: RESOLVED FIXED

(Core :: Audio/Video, defect, P2)

Description

[Pascal Chevrel:pascalc (PTO until April 26)]

STR:

1. Open https://twitter.com/issam_najdawi/status/1506922627075559433
2. Click on the play icon on the video

ER:
Video plays

AR:
The Play icon spins but the video doesn't start

This is reproducible for me on Firefox Linux (wayland and xwayland) on Firefox Nightly (101), beta (100), release (99) and ESR (91).

The video plays correctly on Chrome and Edge. Other videos on twitter play correctly so it's likely to be the video itself that has a problem with Firefox.

Comment 1

[Paul Adenot (:padenot)]

Attachment: nsMediaElement:4,nsMediaElementEvent:4 for the broken video on twitter (see comment 0)

Comment 2

[Paul Adenot (:padenot)]

Attachment: same log, working

It's a fair deal longer. Specifically in the broken case, the ready state doesn't progress to HAVE_ENOUGH_DATA. Looks like this is because we don't have the first frame, something gets stuck.

Comment 3

[Paul Adenot (:padenot)]

https://share.firefox.dev/37FWhUO is a profile of this happening. Not a lot of thing are happening, firefox skips video decoding to the next frame. We don't have markers for decoder init, that's probably a good thing to fix.

There is no RDD in this profile, strangely.

Comment 4

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

This looks similar to bug 1689242. Not sure of the root cause. However, in both cases the video seems to start playing if we call $('video').play().

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 7

[Jim Mathies [:jimm]]

Ashley's going to take a look at this.

Comment 8

[az [:az]]

We took a look at the problem and it appears to be related to the media start times returned by the twitter video. There's a mismatch between the audio / video start times as the audio stream is reported as beginning at 0 seconds with the video starting a few hundred ms after that. With our current logic, this causes our decoder to wait indefinitely on the data needed to render the first frame. We believe that we have some wiggle room here as the spec specifies that streams may start with slightly offset times, with the relevant quote below:

For the purposes of determining if HTMLMediaElement.buffered contains a TimeRanges that includes the current playback position, implementations MAY choose to allow a current playback position at or after presentation start time and before the first TimeRanges to play the first TimeRanges if that TimeRanges starts within a reasonably short time, like 1 second, after presentation start time. This allowance accommodates the reality that muxed streams commonly do not begin all tracks precisely at presentation start time. Implementations MUST report the actual buffered range, regardless of this allowance.

(link to the spec here)

For the example video, Chrome reports 0 from HTMLMediaElement.buffered, whereas Firefox is reporting 0.293522. We're currently working on a fix, possibly to follow the same behavior as Chrome.

Comment 9

[Alastor Wu [:alwu]]

For this issue, there are some perspective we can discuss, (1) proper value of buffered start (2) not be able to access the first video sample. Here are some of my thought.

(1) buffered starting from 0.293522
I think our implementation is correct and chrome is wrong. Based on the HTML spec, buffered represents the range of media resource, which means it should reflect the actual available data range. The MSE spec also mentioned that no matter the allowance for changing current playback position exists or not, buffered MUST report the actual buffered range. As video starts from 0.293522, it makes sense to report the data from there instead of 0 according the MSE spec.

Then, the next steps I kept tracking is this part of spec about buffer monitoring.

If HTMLMediaElement.buffered does not contain a TimeRanges for the current playback position

That leads to another interesting question, what is the current playback position for us? From the HTML spec, it said the current playback in on the media timeline, and the earliest place we can reach on the media timeline is earliest possible position. Although the spec says the earliest position in the stream or resource, it doesn't explicitly mention that if the earliest position should be an intersection of two tracks, or just the earliest position among tracks. If we agree with that the buffered should start from 0.293522, then in our case the earliest possible position and the current playback position should both be 0.293522 as well.

Another question appears, does that mean we should start from 0.293522, even if we already have audio [0, 0.293522]? Or we should still play the resource from 0?

(2) not be able to access the first video sample

This problem is caused by this, because we only allowance the fuzzy range of 250000 and 293522 exceeds the range. The easiest fix would be simply increasing that threshold to 500000, which is the value of EOS_FUZZ.

But I feel this part of code has some defect and we should probably take some time to make it better. First, the comment seems incorrect, during my testing this part of code would only be run in the beginning, not after seeking. If that can be run after seeking, then this part would be definitely wrong because after seeking the data with timestamp 0 won't be guaranteed in the buffered range. Also, based on the definition of Reset(), after calling reset, the demuxer should return the first sample available, not a sample with timestamp 0.

Comment 10

[az [:az]]

Attachment: Bug 1764293 - Separate fuzzing values used for stream start/seek to better support muxed streams with mismatched start times as seen on Twitter/DokiDoki, etc. r=alwu,padenot

Increase playback tolerance for muxed streams with starting times that
differ "within a reasonably short time" as per spec:
https://www.w3.org/TR/media-source-2/#presentation-start-time

In the future, we will want to make additional fixes as per discussion
in the bug thread. Thanks alwu / padenot for the support!

Comment 11

[az [:az]]

Attachment: Bug 1764293 - Add test to validate playback of muxed streams with mismatched start times up to 0.5 seconds. r=alwu,padenot

Depends on D149586

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S4. However, the following bug duplicate has higher severity:

• Bug 1689242: S3

:az, could you consider increasing the severity of this bug to S3?

For more information, please visit auto_nag documentation.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/94eab969df91
Separate fuzzing values used for stream start/seek to better support muxed streams with mismatched start times as seen on Twitter/DokiDoki, etc. r=padenot
https://hg.mozilla.org/integration/autoland/rev/62979ab2a35e
Add test to validate playback of muxed streams with mismatched start times up to 0.5 seconds. r=padenot

Comment 14

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/94eab969df91
https://hg.mozilla.org/mozilla-central/rev/62979ab2a35e