2056945 - Firefox applies a leading empty-edit (elst media_time=-1) presentation offset via MSE but not via progressive <video src> (internally inconsistent)

Status: UNCONFIRMED

(Core :: Audio/Video: Playback, defect, P3)

Description

[Bart[:mormegil]]

Attachment: plain.mp4

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0

Steps to reproduce:

Tested on Firefox 152. Both pages below play the SAME bytes; only the delivery path differs. The fMP4 video track carries one leading empty edit (elst entry: media_time = -1, media_rate = 1, non-zero duration), which offsets video start.

1. MSE path - https://mormegil6.github.io/mse-edit-list-repro/ (video fed via MSE: SourceBuffer.appendBuffer of a DASH init + media segments)
2. Progressive path - https://mormegil6.github.io/mse-edit-list-repro/progressive.html (the identical media as a single file via <video src>)

Actual results:

Firefox handles the same edit two different ways depending on the path:

• MSE: APPLIES the offset. SourceBuffer.buffered.start = 3.0 s, HTMLMediaElement.duration = 8.0 s; picture delayed onto the audio (in sync).
• Progressive <video src>: DROPS it. duration = 5.0 s (media length only), first frame at ~1.0 s (undelayed).

So Firefox disagrees with itself: its MSE path applies a leading empty edit that its progressive path ignores.

Cross-engine context (identical bytes): Chromium and Safari are the reverse - their <video src> applies the offset while their MSE drops it. The Chromium case is reproduced and triaged by Google: https://issues.chromium.org/issues/537235698

Origin: ffmpeg -f dash writes this leading empty edit when the muxed tracks don't start at the same timestamp; the same media served progressively carries the equivalent edit in its moov.

User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0

Expected results:

The two Firefox paths agree on the same edit - ideally both apply the offset, as the MSE path already does (and as Chromium/Safari progressive playback do).

Comment 1

[Jim Mathies [:jimm]]

Not sure how serious this is. Please give this a rating.

Comment 2

[Rob]

We're seeing this in production with MP4s trimmed by ffmpeg (-ss … -c copy), which leaves a leading empty edit on the video track (e.g. segment_duration 1327 in a 1000 movie timescale, video mdhd timescale 15360). Firefox 161 plays these with video ~1.24 s ahead of audio; Chrome is correct.

Is there an ETA for vendoring mp4parse-rust #459 into mozilla-central?

Comment 3

[Bart[:mormegil]]

Possibly related: bug 2082877 (mp4parse-rust #459, merged 27 October), where an edit's segment_duration, which is in the movie timescale per ISO 14496-12, was read in the track's media timescale in the sample-table path for non-fragmented MP4.

The attached plain.mp4 has the same timescale shape as the file Rob describes in comment 2: mvhd timescale 1000, video mdhd timescale 15360, and a leading empty edit of 3000 movie ticks (3.0 s). Read in the video timescale, that edit is 3000/15360, about 0.195 s.

That matches what I now measure on Firefox 159.0.1, which predates the fix. In progressive playback of plain.mp4 the flash that sits at media time 1.0 s appears at 1.195 s (three runs, all 1.195), so the edit is applied but mis-scaled, not ignored as my original report said. The reported duration stays 5.0 s.

The MSE result in this report goes through the fragmented path, which #459 does not touch, and still stands: the offset is applied (buffered.start 3.0 s, duration 8.0 s). The two paths therefore still disagree, but on the progressive side the cause appears to be the timescale bug above, not a missing feature. I will re-run the progressive page once #459 is vendored and report here. plain.mp4 is 11 KB and could serve as a regression file if that helps.

Comment 4

[Bart[:mormegil]]

Correction to my previous comment: the reported duration does not stay at 5.0 s. It reads 5.000 s at load and during playback, and becomes 5.195 s (5.0 + 3000/15360) when playback reaches the end, on Firefox 159.0.1. So the mis-scaled edit also reaches the duration, only late. It is another sign of the timescale bug, not a different behaviour.

Comment 5

[media-triaging-bot]

Thanks for the sample and for the new measurements in comments 3 and 4.

Analysis

The attached plain.mp4 has H.264 video and AAC audio. The movie timescale (mvhd) is 1000, and the video timescale (mdhd) is 15360. The video edit list has an empty edit of 3000 movie ticks (3.0 s), then 5000 ticks of media.

Progressive playback has two problems. Both are in the vendored mp4parse code, which parses non-fragmented MP4.

1. The empty edit has the wrong scale. This is bug 2082877. get_indice_table uses the empty edit duration in movie ticks as if it were in track ticks (lib.rs). So the video moves by 3000/15360 = 0.195 s, not by 3.0 s. The same error explains comment 2: Firefox applies 1327/15360 = 0.086 s in place of 1.327 s, so the video plays 1.24 s early. mp4parse-rust PR 459 fixes this function. It merged on 27 October, but the tree does not have it yet.
2. The duration does not include the empty edit. mp4parse_get_track_info reports the track duration as the edited duration only (lib.rs). For this file that value is 5.0 s, so Firefox reports 5.0 s at load (DecoderData.cpp). At the end of playback, the duration increases to the end position, 5.195 s (comment 4). PR 459 does not change mp4parse_get_track_info, so this problem stays after the vendor update.

The MSE path uses a different parser. Moof::ParseTrun converts the empty edit with the movie timescale (MoofParser.cpp). SourceBuffer then increases the duration to the end of the appended media, 8.0 s (SourceBuffer.cpp). This is why the MSE page shows the 3.0 s offset and the 8.0 s duration.

This is not a regression. The offset code is from the 2017 vendor update in bug 1343793.

Fix direction

The A/V offset needs the vendor update of PR 459, which bug 2082877 tracks. After that update, progressive playback can still report 5.0 s at load and 8.0 s only at the end, while MSE reports 8.0 s. The duration needs a separate change in mp4parse-rust upstream, then a vendor update. One direction is to add the rescaled empty edit to the track duration in mp4parse_get_track_info. Another direction is to use the tkhd duration, which is 8000 movie ticks (8.0 s) for this file.

Caution: the audio duration also goes into the AAC trim calculation in MP4Demuxer.cpp. A change that adds only the empty edit keeps the audio duration of this file the same, because its audio edit has no empty part. With both changes, plain.mp4 should show the flash at 4.0 s and report a duration of 8.0 s at load. plain.mp4 is small and can be the regression test file. Bart plans to test the progressive page again after the vendor update (comment 3).

---

If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.