H.264 video is delayed several seconds compared to Opus audio in some YouTube videos

RESOLVED FIXED

Status

()

P1
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: cers, Assigned: jya)

Tracking

({regression, reproducible})

Trunk
Unspecified
macOS
regression, reproducible
Points:
---

Firefox Tracking Flags

(firefox44+ verified, firefox45+ unaffected, firefox46 unaffected, firefox47 unaffected, relnote-firefox 44+)

Details

(URL)

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
Some (especially long) YouTube videos seem to play the audio offset by several seconds. The quality does not appear to be a factor, nor does jumping in the video change the outcome.

Using mozregression, I found that the bug was introduced in the nightly from 2015-12-04, and the exact regression range is:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=0ea205e989559a0e791a9fdaef35146739466cde&tochange=3021d7cf0c80ef566ef80eea82f43074d2036757

It's a short list, and I suspect bug 1229605 is the culprit.

STR:
1) Load url https://www.youtube.com/watch?v=s8WU1ZouQ6g
2) Wait ~10 seconds (or skip ahead)

Result:
Audio is out of sync with mouth movement

Expected result:
Audio is in sync with mouth movement
(Assignee)

Comment 1

3 years ago
What does "Stats for nerds" content show when you right click on the video?
If you install this plugin: https://github.com/doublec/aboutmedia/raw/master/aboutmedia.xpi
and open a new tab (not a new window) ant type about:media , can you paste here the content?

I can't see sync issue when using full webm (that is the vp9 video and opus audio)

My guess is that the issue with with YouTube encoding, the h264 streams not being synced with their opus audio.
I don't think anyone else is using this configuration, so it's probably not well tested by YouTube.

Chris could you get YouTube to have a look at this stream?
Flags: needinfo?(cpeterson)
(Reporter)

Comment 2

3 years ago
(In reply to Jean-Yves Avenard [:jya] from comment #1)
> What does "Stats for nerds" content show when you right click on the video?

In the nightly from 2015-12-04 I get:
Video ID:
s8WU1ZouQ6g
Dimensions:
854 x 480 * 2
Resolution:
1280 x 720@60
Volume:
100%
Stream Host:
r3---sn-uqj-j2i6
Stream Type:
https
CPN:
8MX6uWlyzfNjpMLB
Mime Type:
video/mp4; codecs="avc1.4d4020"
DASH:
yes (298/251)
Connection Speed:
13664 Kbps
Dropped Frames:
316/2402

and in the nightly from 2015-12-03 I get:
Video ID:
s8WU1ZouQ6g
Dimensions:
854 x 480 * 2
Resolution:
1280 x 720@60
Volume:
100%
Stream Host:
r3---sn-uqj-j2i6
Stream Type:
https
CPN:
-S7SVYc5mLbNgSIs
Mime Type:
video/mp4; codecs="avc1.4d4020"
DASH:
yes (298/140)
Connection Speed:
30061 Kbps
Dropped Frames:
0/2184

(in chrome, the mime type is video/webm; codecs="vp9")

The notable differences are the numbers for DASH and dropped frames.

> If you install this plugin:
> https://github.com/doublec/aboutmedia/raw/master/aboutmedia.xpi
> and open a new tab (not a new window) ant type about:media , can you paste
> here the content?

Except for the header, I get nothing.
(Reporter)

Comment 3

3 years ago
I forgot to mention; installing the addon h264ify seems to resolve whatever issue I am seeing. It doesn't change the mime type reported by "stats for nerds", so I'm not sure what it actually changes...
(Assignee)

Comment 4

3 years ago
stats for nerds only report the info for the video stream

Before, you would have always got h264 video with aac audio, or vp9 video with opus or vorbis audio.

We changed it so that you could use h264 video with opus or vorbis audio.

Chrome always use vp9/opus.

Now what I'm saying is that the combination h264/opus is likely not as well tested by youtube. And I'm fairly certain the issue is in the source content, not in the media player.

Unfortunately, I don't know how to verify this theory as I don't know how I can make youtube serve h264 video with opus audio to Chrome.

JY
Thanks, Christian. The mozregression result is very helpful. I can reproduce this sync problem with this stream when using H.264/Opus but not H.264/AAC or VP9/Opus. I'll follow up with YouTube.
Flags: needinfo?(cpeterson)
Keywords: regression, reproducible
Summary: Video is delayed several seconds compared to audio in some YouTube videos → H.264 video is delayed several seconds compared to Opus audio in some YouTube videos
Jean-Yves says at least three other videos from this same YouTube channel are affected. YouTube confirmed that this is an encoding problem with the media; they can reproduce the problem when using H.264/Opus in Chrome (if they change Chrome's default settings).

Until we know how widespread this problem is we should consider flipping the "media.mediasource.webm.audio.enabled" pref to false (thus undoing bug 1229605).
Blocks: 1229605
Christian, could you please confirm that setting "media.mediasource.webm.audio.enabled" pref in about:config to false, fixes the out of sync playback issues? Thanks!
Flags: needinfo?(cers)
(Assignee)

Comment 8

3 years ago
(In reply to Ritu Kothari (:ritu) from comment #7)
> Christian, could you please confirm that setting
> "media.mediasource.webm.audio.enabled" pref in about:config to false, fixes
> the out of sync playback issues? Thanks!

That sure will..

However note that as of now, I can't reproduce the issue with this video, so setting that pref on or off won't be able to guarantee this problem is gone.

YouTube/Google has found what the problem is in their encoding. I would be surprise if they were that quick to fix it.. but you never know
(Assignee)

Comment 9

3 years ago
YouTube has broken encoding when we enable this codec combination.
Attachment #8716127 - Flags: review?(cpeterson)
(Reporter)

Comment 10

3 years ago
(In reply to Ritu Kothari (:ritu) from comment #7)
> Christian, could you please confirm that setting
> "media.mediasource.webm.audio.enabled" pref in about:config to false, fixes
> the out of sync playback issues? Thanks!

It does indeed. It appears the problem has gone away on that particular video in the mean time though, but the next one in the series still has it: https://www.youtube.com/watch?v=zetif7-zxs4

Flipping the pref fixes it there though.
Flags: needinfo?(cers)
Comment on attachment 8716127 [details] [diff] [review]
Disable opus/vorbis audio with h264.

Review of attachment 8716127 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM
Attachment #8716127 - Flags: review?(cpeterson) → review+
[Tracking Requested - why for this release]:

This is a regression in Firefox 44 from bug 1229605. The problem is technically a YouTube encoding bug, but Firefox is the only affected browser because we started using the uncommon combination of H.264 video with Opus audio. Chrome uses VP9/Opus and other browsers use H.264/AAC.
status-firefox44: --- → affected
status-firefox45: --- → affected
status-firefox46: --- → affected
tracking-firefox44: --- → ?
tracking-firefox45: --- → ?
YouTube has fixed the encoding of the original video, but I still see A/V sync problems with other videos from the same channel when using H.264 and Opus:

https://www.youtube.com/watch?v=qnSZJmYG-Wc&t=6m
Comment on attachment 8716127 [details] [diff] [review]
Disable opus/vorbis audio with h264.

Approval Request Comment
[Feature/regressing bug #]: bug 1229605
[User impact if declined]: Some YouTube videos will have audio and video that is out-of-sync by multiple seconds. We don't know how many videos may be affected.
[Describe test coverage new/current, TreeHerder]:
[Risks and why]: This should be a safe fix because we will be reverting to our known-good code path (AAC audio), which has been the Firefox default for a long time.
[String/UUID change made/needed]: No changes
Attachment #8716127 - Flags: approval-mozilla-release?
(In reply to Chris Peterson [:cpeterson] from comment #12)
>
> .......The problem is
> technically a YouTube encoding bug, but Firefox is the only affected browser
> because we started using the uncommon combination of H.264 video with Opus
> audio. Chrome uses VP9/Opus and other browsers use H.264/AAC.

Hi Chris, Jean-Yves: If we believe that this uncommon combination is something that YouTube team is not committed to support fully (perhaps lower in their list of priorities), should we reconsider our decision to go with H.264 + Opus? And we choose something more standard which will be less likely to break on their side.
Flags: needinfo?(jyavenard)
Flags: needinfo?(cpeterson)
Comment on attachment 8716127 [details] [diff] [review]
Disable opus/vorbis audio with h264.

A/V component team (ChrisP) has mentioned this as a medium severity issue causing audio-video playback getting out of sync (4-10 secs). The risk associated with taking this fix is minimal, it will now use AAC audio codec instead of Opus. Taking this another ride-along in 44.0.1.
Attachment #8716127 - Flags: approval-mozilla-release? → approval-mozilla-release+
You'll need to request approval for aurora/beta before this can land there to make sure this doesn't regress.

Comment 18

3 years ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-release/rev/eaea4914dc83
status-firefox44: affected → fixed
Thanks, Wes. We plan to watch this bug in the pre-release channels a little longer to understand how widespread the problem really is.
Flags: needinfo?(cpeterson)
(Assignee)

Comment 20

3 years ago
Only applying this to release, will allow YouTube to fix all those videos. They have mentioned that they are putting in place a mechanism to automatically detect them
Flags: needinfo?(jyavenard)
Reproduced the initial issue using Firefox 44.0 RC and 45 beta . Was unable to reproduce on Firefox 44.0.1 build 2 RC on Windows 7 64-bit, Mac OS X 10.9.5, Windows 10 64-bit. 
I was able to reproduced once out of 5 tries on Ubuntu 14.04 64 with 44.0.1 build 2, but failed to reproduce with another profile after that. I used a clean profile every time I tested and checked that "media.mediasource.webm.audio.enabled" was set to "false". Steps were basically jump from one long video to another, drag tabs out of window, reattach, mute videos, unmute - not exact steps I know.
Any ideas?
Flags: needinfo?(jyavenard)
Oh I forgot to say that I encountered that on this link: https://www.youtube.com/watch?v=EmKGAcPGPdQ, I did visit other channels that had long videos as well but they worked ok.
(Assignee)

Comment 23

3 years ago
You will not be able to reproduce the problem on Linux (or Windows XP for that matter). We enable MSE/webm when either h264 isn't available or if hardware decoding isn't available.
There's no hardware decoding available on Linux; as such you will always get MSE/webm for both audio and video.
The issue will only happen if YouTube serves us a mp4/h264 video stream with a webm/opus audio stream.

You can see what video type is being used by right-clicking in the player and selecting "Stats for Nerds"
Flags: needinfo?(jyavenard)
(Assignee)

Comment 24

3 years ago
Also note that YouTube has manually fixed all the videos mentioned in this thread. So verifying that the problem is gone is going to be difficult now.
Then I think that on Ubuntu I hit into something else, maybe some skipped frames as you said or something else. 
I think we can mark this as verified on Firefox 44 since we did not encountered the delay on this youtube links from the bug and others from that channel and others.
status-firefox44: fixed → verified
"Disable opus/vorbis audio with H.264 (1245696)" added to the release notes
tracking-firefox44: ? → +
tracking-firefox45: ? → +
relnote-firefox: --- → 44+
Just for future visitors to this bug - that patch description / release note description is a bit misleading - we're disabling all WebM MSE support when the browser has access to a H.264 decoder. This is not ideal - e.g. other sites might want to do WebM audio streaming only, or might have working H.264/Opus encodes.

This bug is only a temporary fix - real fixes include Youtube fixing all of their encodings, and/or enabling VP9 everywhere.
Jean-Yves, should we take this patch in 45 too? Thanks
Assignee: nobody → jyavenard
Flags: needinfo?(jyavenard)
(Assignee)

Comment 29

3 years ago
YouTube has stated that this was a one off and was to implement a global solution to detect and fix those bad videos.
So the intent was to allow them to fix it and only disable it in 45 if it wasn't done on time

So I'll leave Chris Peterson to answer this question...
Flags: needinfo?(jyavenard) → needinfo?(cpeterson)
(In reply to Sylvestre Ledru [:sylvestre] from comment #28)
> Jean-Yves, should we take this patch in 45 too? Thanks

No. We would like to try to ship H.264+Opus again, so I don't think we want this patch in 45.

As Jean-Yves said, this issue was (apparently) an isolated YouTube encoding problem. We disabled the support for H.264+Opus in 44.0.1, so we had time to confirm the scope of the YouTube problem.

I'm closing this bug as FIXED because YouTube re-encoded the known bad videos.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
status-firefox45: affected → unaffected
status-firefox46: affected → unaffected
status-firefox47: affected → unaffected
Flags: needinfo?(cpeterson)
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1249625
You need to log in before you can comment on or make changes to this bug.