1245696 - H.264 video is delayed several seconds compared to Opus audio in some YouTube videos

Status: RESOLVED FIXED

(Core :: Audio/Video, defect, P1)

Description

[Christian Sonne [:cers]]

Some (especially long) YouTube videos seem to play the audio offset by several seconds. The quality does not appear to be a factor, nor does jumping in the video change the outcome.

Using mozregression, I found that the bug was introduced in the nightly from 2015-12-04, and the exact regression range is:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=0ea205e989559a0e791a9fdaef35146739466cde&tochange=3021d7cf0c80ef566ef80eea82f43074d2036757

It's a short list, and I suspect bug 1229605 is the culprit.

STR:
1) Load url https://www.youtube.com/watch?v=s8WU1ZouQ6g
2) Wait ~10 seconds (or skip ahead)

Result:
Audio is out of sync with mouth movement

Expected result:
Audio is in sync with mouth movement

Comment 1

[Jean-Yves Avenard [:jya]]

What does "Stats for nerds" content show when you right click on the video?
If you install this plugin: https://github.com/doublec/aboutmedia/raw/master/aboutmedia.xpi
and open a new tab (not a new window) ant type about:media , can you paste here the content?

I can't see sync issue when using full webm (that is the vp9 video and opus audio)

My guess is that the issue with with YouTube encoding, the h264 streams not being synced with their opus audio.
I don't think anyone else is using this configuration, so it's probably not well tested by YouTube.

Chris could you get YouTube to have a look at this stream?

Comment 2

[Christian Sonne [:cers]]

(In reply to Jean-Yves Avenard [:jya] from comment #1)
> What does "Stats for nerds" content show when you right click on the video?

In the nightly from 2015-12-04 I get:
Video ID:
s8WU1ZouQ6g
Dimensions:
854 x 480 * 2
Resolution:
1280 x 720@60
Volume:
100%
Stream Host:
r3---sn-uqj-j2i6
Stream Type:
https
CPN:
8MX6uWlyzfNjpMLB
Mime Type:
video/mp4; codecs="avc1.4d4020"
DASH:
yes (298/251)
Connection Speed:
13664 Kbps
Dropped Frames:
316/2402

and in the nightly from 2015-12-03 I get:
Video ID:
s8WU1ZouQ6g
Dimensions:
854 x 480 * 2
Resolution:
1280 x 720@60
Volume:
100%
Stream Host:
r3---sn-uqj-j2i6
Stream Type:
https
CPN:
-S7SVYc5mLbNgSIs
Mime Type:
video/mp4; codecs="avc1.4d4020"
DASH:
yes (298/140)
Connection Speed:
30061 Kbps
Dropped Frames:
0/2184

(in chrome, the mime type is video/webm; codecs="vp9")

The notable differences are the numbers for DASH and dropped frames.

> If you install this plugin:
> https://github.com/doublec/aboutmedia/raw/master/aboutmedia.xpi
> and open a new tab (not a new window) ant type about:media , can you paste
> here the content?

Except for the header, I get nothing.

Comment 3

[Christian Sonne [:cers]]

I forgot to mention; installing the addon h264ify seems to resolve whatever issue I am seeing. It doesn't change the mime type reported by "stats for nerds", so I'm not sure what it actually changes...

Comment 4

[Jean-Yves Avenard [:jya]]

stats for nerds only report the info for the video stream

Before, you would have always got h264 video with aac audio, or vp9 video with opus or vorbis audio.

We changed it so that you could use h264 video with opus or vorbis audio.

Chrome always use vp9/opus.

Now what I'm saying is that the combination h264/opus is likely not as well tested by youtube. And I'm fairly certain the issue is in the source content, not in the media player.

Unfortunately, I don't know how to verify this theory as I don't know how I can make youtube serve h264 video with opus audio to Chrome.

JY

Comment 5

[Chris Peterson [:cpeterson]]

Thanks, Christian. The mozregression result is very helpful. I can reproduce this sync problem with this stream when using H.264/Opus but not H.264/AAC or VP9/Opus. I'll follow up with YouTube.

Comment 6

[Chris Peterson [:cpeterson]]

Jean-Yves says at least three other videos from this same YouTube channel are affected. YouTube confirmed that this is an encoding problem with the media; they can reproduce the problem when using H.264/Opus in Chrome (if they change Chrome's default settings).

Until we know how widespread this problem is we should consider flipping the "media.mediasource.webm.audio.enabled" pref to false (thus undoing bug 1229605).

Comment 7

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Christian, could you please confirm that setting "media.mediasource.webm.audio.enabled" pref in about:config to false, fixes the out of sync playback issues? Thanks!

Comment 8

[Jean-Yves Avenard [:jya]]

(In reply to Ritu Kothari (:ritu) from comment #7)
> Christian, could you please confirm that setting
> "media.mediasource.webm.audio.enabled" pref in about:config to false, fixes
> the out of sync playback issues? Thanks!

That sure will..

However note that as of now, I can't reproduce the issue with this video, so setting that pref on or off won't be able to guarantee this problem is gone.

YouTube/Google has found what the problem is in their encoding. I would be surprise if they were that quick to fix it.. but you never know

Comment 9

[Jean-Yves Avenard [:jya]]

Attachment: Disable opus/vorbis audio with h264.

YouTube has broken encoding when we enable this codec combination.

Comment 10

[Christian Sonne [:cers]]

(In reply to Ritu Kothari (:ritu) from comment #7)
> Christian, could you please confirm that setting
> "media.mediasource.webm.audio.enabled" pref in about:config to false, fixes
> the out of sync playback issues? Thanks!

It does indeed. It appears the problem has gone away on that particular video in the mean time though, but the next one in the series still has it: https://www.youtube.com/watch?v=zetif7-zxs4

Flipping the pref fixes it there though.

Comment 11

[Chris Peterson [:cpeterson]]

Comment on attachment 8716127 [details] [diff] [review]
Disable opus/vorbis audio with h264.

Review of attachment 8716127 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM

Comment 12

[Chris Peterson [:cpeterson]]

[Tracking Requested - why for this release]:

This is a regression in Firefox 44 from bug 1229605. The problem is technically a YouTube encoding bug, but Firefox is the only affected browser because we started using the uncommon combination of H.264 video with Opus audio. Chrome uses VP9/Opus and other browsers use H.264/AAC.

Comment 13

[Chris Peterson [:cpeterson]]

YouTube has fixed the encoding of the original video, but I still see A/V sync problems with other videos from the same channel when using H.264 and Opus:

https://www.youtube.com/watch?v=qnSZJmYG-Wc&t=6m

Comment 14

[Chris Peterson [:cpeterson]]

Comment on attachment 8716127 [details] [diff] [review]
Disable opus/vorbis audio with h264.

Approval Request Comment
[Feature/regressing bug #]: bug 1229605
[User impact if declined]: Some YouTube videos will have audio and video that is out-of-sync by multiple seconds. We don't know how many videos may be affected.
[Describe test coverage new/current, TreeHerder]:
[Risks and why]: This should be a safe fix because we will be reverting to our known-good code path (AAC audio), which has been the Firefox default for a long time.
[String/UUID change made/needed]: No changes

Comment 15

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

(In reply to Chris Peterson [:cpeterson] from comment #12)
>
> .......The problem is
> technically a YouTube encoding bug, but Firefox is the only affected browser
> because we started using the uncommon combination of H.264 video with Opus
> audio. Chrome uses VP9/Opus and other browsers use H.264/AAC.

Hi Chris, Jean-Yves: If we believe that this uncommon combination is something that YouTube team is not committed to support fully (perhaps lower in their list of priorities), should we reconsider our decision to go with H.264 + Opus? And we choose something more standard which will be less likely to break on their side.

Comment 16

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8716127 [details] [diff] [review]
Disable opus/vorbis audio with h264.

A/V component team (ChrisP) has mentioned this as a medium severity issue causing audio-video playback getting out of sync (4-10 secs). The risk associated with taking this fix is minimal, it will now use AAC audio codec instead of Opus. Taking this another ride-along in 44.0.1.

Comment 17

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

You'll need to request approval for aurora/beta before this can land there to make sure this doesn't regress.

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-release/rev/eaea4914dc83

Comment 19

[Chris Peterson [:cpeterson]]

Thanks, Wes. We plan to watch this bug in the pre-release channels a little longer to understand how widespread the problem really is.

Comment 20

[Jean-Yves Avenard [:jya]]

Only applying this to release, will allow YouTube to fix all those videos. They have mentioned that they are putting in place a mechanism to automatically detect them

Comment 21

[Bogdan Maris, Desktop Test Engineering]

Reproduced the initial issue using Firefox 44.0 RC and 45 beta . Was unable to reproduce on Firefox 44.0.1 build 2 RC on Windows 7 64-bit, Mac OS X 10.9.5, Windows 10 64-bit. 
I was able to reproduced once out of 5 tries on Ubuntu 14.04 64 with 44.0.1 build 2, but failed to reproduce with another profile after that. I used a clean profile every time I tested and checked that "media.mediasource.webm.audio.enabled" was set to "false". Steps were basically jump from one long video to another, drag tabs out of window, reattach, mute videos, unmute - not exact steps I know.
Any ideas?

Comment 22

[Bogdan Maris, Desktop Test Engineering]

Oh I forgot to say that I encountered that on this link: https://www.youtube.com/watch?v=EmKGAcPGPdQ, I did visit other channels that had long videos as well but they worked ok.

Comment 23

[Jean-Yves Avenard [:jya]]

You will not be able to reproduce the problem on Linux (or Windows XP for that matter). We enable MSE/webm when either h264 isn't available or if hardware decoding isn't available.
There's no hardware decoding available on Linux; as such you will always get MSE/webm for both audio and video.
The issue will only happen if YouTube serves us a mp4/h264 video stream with a webm/opus audio stream.

You can see what video type is being used by right-clicking in the player and selecting "Stats for Nerds"

Comment 24

[Jean-Yves Avenard [:jya]]

Also note that YouTube has manually fixed all the videos mentioned in this thread. So verifying that the problem is gone is going to be difficult now.

Comment 25

[Bogdan Maris, Desktop Test Engineering]

Then I think that on Ubuntu I hit into something else, maybe some skipped frames as you said or something else. 
I think we can mark this as verified on Firefox 44 since we did not encountered the delay on this youtube links from the bug and others from that channel and others.

Comment 26

[Sylvestre Ledru [:Sylvestre]]

"Disable opus/vorbis audio with H.264 (1245696)" added to the release notes

Comment 27

[Thomas Daede [:TD-Linux]]

Just for future visitors to this bug - that patch description / release note description is a bit misleading - we're disabling all WebM MSE support when the browser has access to a H.264 decoder. This is not ideal - e.g. other sites might want to do WebM audio streaming only, or might have working H.264/Opus encodes.

This bug is only a temporary fix - real fixes include Youtube fixing all of their encodings, and/or enabling VP9 everywhere.

Comment 28

[Sylvestre Ledru [:Sylvestre]]

Jean-Yves, should we take this patch in 45 too? Thanks

Comment 29

[Jean-Yves Avenard [:jya]]

YouTube has stated that this was a one off and was to implement a global solution to detect and fix those bad videos.
So the intent was to allow them to fix it and only disable it in 45 if it wasn't done on time

So I'll leave Chris Peterson to answer this question...

Comment 30

[Chris Peterson [:cpeterson]]

(In reply to Sylvestre Ledru [:sylvestre] from comment #28)
> Jean-Yves, should we take this patch in 45 too? Thanks

No. We would like to try to ship H.264+Opus again, so I don't think we want this patch in 45.

As Jean-Yves said, this issue was (apparently) an isolated YouTube encoding problem. We disabled the support for H.264+Opus in 44.0.1, so we had time to confirm the scope of the YouTube problem.

I'm closing this bug as FIXED because YouTube re-encoded the known bad videos.