1776346 - Regression in version 91: Media playback not working with blob urls

Status: RESOLVED INVALID

(Core :: Audio/Video, defect, P3)

Description

[Shiva]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36

Steps to reproduce:

The problem can be viewed at:
https://cb-mozilla-bug.s3.ap-south-1.amazonaws.com/index.html

• "Play Thumbnail" does not work from Firefox 91 onwards. It is working fine on Firefox 90 and below. It also works fine on Chrome.

Steps:

1. Create a mp4 file by adding init and data segments (init.mp4 + segment.mp4).
2. Create a blob url with this data.
3. Set this to a <video> element' src, in order to playback.

Snippet:
const initResponse = await axios.get(initUrl, { responseType: "arraybuffer" });
const dataResponse = await axios.get(dataUrl, { responseType: "arraybuffer" });
const finalData = appendBuffer(initResponse.data, dataResponse.data);

    const video = document.querySelector("video");
    let blob = new Blob([finalData], { type: "video/mp4" });
    video.src = URL.createObjectURL(blob);
    video.load();
    video.loop = true;
    video.play();

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:101.0) Gecko/20100101 Firefox/101.0

Actual results:

Playback did not happen.

Expected results:

The video contains a single video frame (thumbnail), and is supposed to be displayed in the video element.

Comment 1

[Shiva]

This seems to be similar to this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1722253

The playback worked fine on 90.x, but is broken on 91.x and later versions.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[Alice0775 White]

Regression window:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=d337892115ed05eabef76f250fc8f794d26b81d6&tochange=5f7a9f34ce185a1dc0f6ab4453251c45ba3728f4

Comment 4

[Shiva]

Hi Team, could you please confirm if this is indeed a regression?

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1718709

Comment 6

[Shiva]

An observation:
The asset is encoded using 0.17fps.
If the same asset is encoded using 25fps, the issue is not seen.

Comment 7

[Shiva]

Team, what is the expected timeline to fix this issue?

Comment 8

[Dianna Smith [:diannaS]]

:padenot Do you know what the Priority/severity could be for this? Its a wont fix for 104, but in order to answer #c7, i think we need more information on impact. ty!

Comment 9

[Paul Adenot (:padenot)]

Alastor, you're in the range, can you have a look?

Comment 10

[Paul Adenot (:padenot)]

This seems to work as intended here on macOS.

Comment 11

[Shiva]

It works on Mac, the issue is only on Windows.

Comment 12

[Shiva]

Folks, could you please let us know if this would be fixed in upcoming versions?
Do let me know if you need any further information to analyze the issue.

Comment 13

[Alastor Wu [:alwu]]

This issue seems another WMF decoder issue. In bug 1718709, we encountered an issue where WMF decoder would return an incorrect first frame to us and we didn't want that frame because if that incorrect video frame gets stored in our video queue, which would cause MDSM triggering the mechanism to request the next key frame.

For that Thumbnail video, the first video sample we feed to the WMF decoder has pts 8080000 dts 8000000 and duration 6000000. The sample WMF returns to us is pts 0 duration 6024096, which is apparently different from the one we sent before. WMF decoder seems adjusting the timestamp for the first video frame, but it didn't adjust it correctly.

Same situation happens on another long video, the first sample WMF decoder is also from timestamp 0, but all other following samples follows the original timestamp, which means WMF decoder only adjusts the first sample (and also didn't adjust it perfectly.....)

On MacOS, AppleVTDecoder returns the data as our expectation, it doesn't tamper the data, so that video can be played correctly on MacOS. (As we would adjust video's start time by ourselves later if the video doesn't start from 0) Chrome and Edge use DXVA decoder directly, so they won't encounter this issue as well.

Maybe we need to add more workaround for WMF decoder, which is not ideal and nothing I would like to avoid.... but I don't have a very good idea about how to fix it now. I need to think about it.

Comment 14

[Shiva]

We investigated further on this issue.

original mp4 (with issue) has the following values

• tfdt: base_media_decode_time: 720000 (90Khz)
• composition time offset: 7200 (90khz)
• DTS: 720000
• CTS (composition timestamp): 727200

We created a new mp4 by manually updating these values as follows:

• tfdt: base_media_decode_time: 0
• composition time offset: 0
• DTS: 0
• CTS (composition timestamp): 0

With these values the issue is not seen and Firefox is able to display the thumbnail successfully.

Note that changing only the tfdt or the composition timeoffset is not enough. Both changes are mandatory to make the segment playable. So it looks like Firefox is not able to play assets that do not start with PTS=0.

The original and modified mp4 are available here:

• original (not working): https://cb-mozilla-bug.s3.ap-south-1.amazonaws.com/thumbnail-original.mp4
• modified (working): https://cb-mozilla-bug.s3.ap-south-1.amazonaws.com/thumbnail-Modified.mp4

We have done the above only to understand more about the issue. We cannot deploy the solution to production environment because this segment comes from a DASH manifest, and the standards explicitly require that the segment's CTS corresponds to the manifest timestamp.

This has to be addressed from the browser.

Comment 15

[Alastor Wu [:alwu]]

To be clear, this issue is not on your side. The video you sent to us is not perfect (not from zero), but we should still be able to play that. We can definitely play video with PTS != 0 (like another video in your test case, which is also not starting from zero).

This issue now is because that video only contains one video frame, which hits the internal issue of the media foundation. We expect it to return the sample which has exact timestamp as what we gave it, but it returned an adjusted sample which has different timestamp and duration. (But it will return correct samples since the second frame, which explains why another video works)

The simplest fix from your side would be containing at least two frames (although it's still not ideal) And the same time we will keep working on this and see how we can avoid the incorrect first sample. In addition, this is Windows only issue, on other platforms, you should be able to play the original thumbnail video.

Comment 16

[Alastor Wu [:alwu]]

The link in the comment0 is already invalid, so close this bug. But feel to reopen this bug if anyone has a new link which can be used to reproduce this issue. Thanks!