776137 - Video and Sound (Youtube) asynchronous if AirPlay is enabled

Status: RESOLVED FIXED

(Core :: Audio/Video: cubeb, defect, P2)

Description

[Nomis101]

If you enable AirPlay on Mountain Lion, than the sound has a delay of a few seconds (I think because of error correction). Apple has solved this for Quicktime and Quicklook by also delaying the video the same time. This also works in Safari 6 if you watch Youtube videos. So the sound and the video is synchronous again if AirPlay is enabled (works for Youtube, but not for other video sites). If you watch Youtube videos in Firefox, than this doesn't work and Video and Sound is asynchronous.

10.8 12A269

Comment 1

[Matthew Gregan [:kinetik]]

Thanks for the report.  We can fix this for HTML 5 video if that's where the issue lies, but Flash's audio is out of our control and will require an update from Adobe if you're seeing this problem with Flash video playback.

Anyway, this isn't too surprising for our side; the way we calculate the audio playback position depends on the audio callback running frequently with low latency (see http://mxr.mozilla.org/mozilla-central/source/media/libsydneyaudio/src/sydney_audio_mac.c#431).  We're using that technique to work around issues we had with CoreAudio's clock reporting bad values on suspend/resume and device changes.

Fixing this correctly probably means moving back to using CoreAudio's clock, which'll mean we need to find another way to address the other issues.

Comment 2

[Nomis101]

Exactly, sorry I haven't mentioned this. In Safari I use the YouTube5 extension, so the videos in Youtube are all HTML5. If I switch to Flash, than it's also asynchronous in Safari. In Firefox both, HTML5 and Flash, is asynchronous.

Comment 3

[Cezar „ikari” Pokorski]

Fixing this would be appreciated. It's annoying to have to use another browser for youtube links :)

Comment 4

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to contact from comment #3)
> Fixing this would be appreciated. It's annoying to have to use another
> browser for youtube links :)

Is this issue still reproducible? Is the audio early or late?

Comment 5

[Cezar „ikari” Pokorski]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #4)
> Is this issue still reproducible? Is the audio early or late?

If reproducible (I can't check at the moment, not having an AirPlay device nearby), the sound was late. Basically it seemed like Firefox did not do what The Other Browser does here - it holds the video playback for a few seconds to sync it with the output audio buffer of the audio device. In The Other Browser it works with video only (not interactive content), which would support the observation that it's about delaying the video rendering to sync with audio. 
In Firefox, last time I checked, simply audio and video start their playback together, but when streaming through AirPlay, audio gets a delay.

Comment 6

[Nomis101]

(In reply to contact from comment #5)
> In Firefox, last time I checked, simply audio and video start their playback
> together, but when streaming through AirPlay, audio gets a delay.

I checked just know with latest Nightly. And this is still the case. Over Airplay the sound is coming a few seconds too late.

Comment 7

[Matthew Gregan [:kinetik]]

The stream clock in cubeb_audiounit.c (audiounit_stream_get_position) doesn't adjust for hardware or playout latency.  We have the code (added much later) in audiostream_stream_get_latency to query the hardware latency and the playout latency looks trivial to calculate, so it would be fairly simple to fix this.

Comment 8

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Blake - do you think this would be a good fit for Chun Min after he's finished with 5.1 stuff?

Comment 9

[Matthew Gregan [:kinetik]]

(In reply to Matthew Gregan [:kinetik] from comment #7)
> The stream clock in cubeb_audiounit.c (audiounit_stream_get_position)
> doesn't adjust for hardware or playout latency.  We have the code (added
> much later) in audiostream_stream_get_latency to query the hardware latency
> and the playout latency looks trivial to calculate, so it would be fairly
> simple to fix this.

It turns out this doesn't work as assumed, so there wasn't a quick fix for this.  The playout latency reported by audiostream_stream_get_latency doesn't seem to include delay induced by AirPlay (although it should be correctly queried via the device_latency_frames request, possibly a bug there).  Looking around at the time, it looked like the solution was to register notifications to detect when the audio device switched to and from an AirPlay device (which we now have most/all of the code for) and then add a fixed 2000ms offset to the clock to repair A/V sync.

Some history of VLC trying to fix this: https://trac.videolan.org/vlc/ticket/7127
And eventually reverting the changes due to problems: https://trac.videolan.org/vlc/ticket/14042

Comment 10

[Blake Wu [:bwu][:blakewu]]

I can see this problem on my macOS Sierra but audio comes earlier than video. FWIW, Chrome doesn't have this problem. 

It is an very interesting bug. Yeah, it is a good bug for Chunmin to check and know more about cubeb. He should be fascinated to this. 

Matthew,
Do you mind letting Chunmin to check this bug?

Comment 11

[C.M.Chang[:chunmin]]

I tested with AppleTV 3 in the meeting room but it works fine. The audio and video are synchronous. Blake will help us to test with his Apple TV 2 and see what could we find.

Comment 12

[Blake Wu [:bwu][:blakewu]]

I tried the latest nightly with my Apple TV 3 again. And now I cannot see AV unsync problem. It looks like something magically changed on the nightly and it fixed this problem somehow. 

ChunMin, 
BTW, I was wrong about the version[1] of Apple TV in our office. It should be Apple TV "4".  

[1]https://support.apple.com/en-us/HT200008

Comment 13

[samer]

I still have a 2 second latency between audio and video when connected to a speaker via AirPlay.
I'm running on OSX 10.12.3, tested on both FF 56.0a1 and 54.0.1
Can you please re-open this bug?

Comment 14

[Matthew Gregan [:kinetik]]

(In reply to samer from comment #13)
> I still have a 2 second latency between audio and video when connected to a
> speaker via AirPlay.
> I'm running on OSX 10.12.3, tested on both FF 56.0a1 and 54.0.1
> Can you please re-open this bug?

Thanks for the report.  Which device and version are you using at the output side, an Apple TV (which version?) or something else?

Comment 15

[samer]

Sorry, I should have mentioned that I'm using a Raspberry Pi running Shairport-Sync[0]. I have previously filed a bug report[1] to the maintainer, but he claims that the issue is particular to Firefox:

> There is always some delay when you send audio to an Airplay device such as Shairport Sync or the Apple Airport Express. The delay is set by the audio sender and is in the range of about 400 milliseconds to over two seconds. Most senders set the delay at two seconds, including the Mac, when you're routing system audio to an AirPlay device. To compensate for the delay, Safari and Chrome and the Quicktime Player delay video part of a movie by two seconds so that the video and audio are in sync. Firefox doesn't do it. Also, Safari and Chrome don't do it for all YouTube content.\

I can confirm that I am only facing the problem with Firefox, not Safari or Chrome.

[0] https://github.com/mikebrady/shairport-sync/
[1] https://github.com/mikebrady/shairport-sync/issues/161)

Comment 16

[u598460]

I have the same issue. Playing a youtube video with Firefox results in audio and video being out of sync. Chrome and Safari play the same youtube video just fine with audio matching the video.

I'm on a MacBook Pro 5,3 with OS X 10.11.6 and Firefox 54.0.1 (64-bit). I use an Apple Airport Express (1st Generation, v7.6.8) as my Wi-Fi router and to stream audio via Airplay.

Steps to reproduce:

1. Connect to Airport Express
2. Set the Audio Output Device to the Airplay device
3. Open a youtube video

Expected result: Video plays with audio and video in sync.
Actual result: Video plays, but audio lags approximately 2 seconds behind.

Comment 17

[Bulk Bug Changes for mreavy's org]

Mass change P1->P2 to align with new Mozilla triage process

Comment 18

[Blake Wu [:bwu][:blakewu]]

Hi Reporters, 
Can you help test with Firefox 57 to see if this is still a problem or not?
Thanks.

Comment 19

[u598460]

(In reply to Blake Wu [:bwu][:blakewu] from comment #18)
> Hi Reporters, 
> Can you help test with Firefox 57 to see if this is still a problem or not?
> Thanks.

Unfortunately I have no access to my Airport Express at the moment, so I cannot test.

Comment 20

[samer]

The problem persists for me when running Firefox 57 beta.
MacOS 10.12 + Raspberry Pi running Shairport-sync.

Comment 21

[Blake Wu [:bwu][:blakewu]]

(In reply to samer from comment #20)
> The problem persists for me when running Firefox 57 beta.
> MacOS 10.12 + Raspberry Pi running Shairport-sync.
I cannot repro this bug with the Firefox 57.0b11 and the latest nightly 58 on macOS 10.13 (17A405) with Apple TV (3rd generation). 

What is your Apple TV version? 
https://support.apple.com/en-us/HT200008

Comment 22

[samer]

(In reply to Blake Wu [:bwu][:blakewu] from comment #21)
> (In reply to samer from comment #20)
> > The problem persists for me when running Firefox 57 beta.
> > MacOS 10.12 + Raspberry Pi running Shairport-sync.
> I cannot repro this bug with the Firefox 57.0b11 and the latest nightly 58
> on macOS 10.13 (17A405) with Apple TV (3rd generation). 
> 
> What is your Apple TV version? 
> https://support.apple.com/en-us/HT200008

As mentioned above, I'm emulating an Apple TV using a Raspberry Pi and Shairport-sync (https://github.com/mikebrady/shairport-sync)

Comment 23

[Cezar „ikari” Pokorski]

Problem still persists with Firefox 56 and Denon AVR X2000 receiver.

Comment 24

[Blake Wu [:bwu][:blakewu]]

If now AV unsync issue only happens on Shairport-sync, not Apple TV, it would be better to file another bug and close this one.

Comment 25

[Cezar „ikari” Pokorski]

(In reply to Blake Wu [:bwu][:blakewu] from comment #24)
> If now AV unsync issue only happens on Shairport-sync, not Apple TV, it
> would be better to file another bug and close this one.

As I mentioned, not only on Shairport-sync, but also on AirPlay compatible amplifiers.

Comment 26

[Ben S]

Can confirm also experiencing audio latency when using airplay.

Firefox: 58.0b7 (64-bit)
MacOS 10.13.1
Airplay Audio Device: Sony Speaker RDP-XA900iP

Latency does not exist when using Safari (Version 11.0.1 (13604.3.5)) or other applications like iTunes.

Comment 27

[asesg4]

The problem persists with Firefox 59.0.1 on macOS 10.13.3. There extists a fixed 2 second audio delay when airplaying to Apple HomePod. The problem doesn't affect Safari or Chrome.

For your reference this is the chromium bug regarding this problem: https://bugs.chromium.org/p/chromium/issues/detail?id=783282

Comment 28

[Matthew Gregan [:kinetik]]

(In reply to asesg4 from comment #27)
> For your reference this is the chromium bug regarding this problem:
> https://bugs.chromium.org/p/chromium/issues/detail?id=783282

Thanks, that's useful.  We've got code that queries kAudioDevicePropertyLatency for device enumeration, but we're not using it for latency adjustment.  From a quick test with my ATV4, I get surprisingly low values (~150ms)... I'll experiment a bit more and see if I can get it working correctly.

Comment 29

[Matthew Gregan [:kinetik]]

Nightly test build here: https://index.taskcluster.net/v1/task/gecko.v2.try.revision.d7512296fab96bae8ab91979bca41ae8b892db62.firefox.macosx64-opt/artifacts/public/build/target.dmg

This looks and sounds right for local playback and using AirPlay to an Apple TV 4 (which is reporting ~150ms).  I'd appreciate feedback on how well this works for others, particular anyone with device that has a ~2s delay.

Comment 30

[Paul Adenot (:padenot)]

NI to myself so I test with my Bluetooth setup at home that has high latency.

Comment 31

[asesg4]

The issue is fixed in the test build. Audio and video now syncs on my HomePod. Thanks.

Comment 32

[samer]

Nightly works for me!
Emulating airplay using a Raspberry Pi with Shairport-Sync.

Thank you so much :kinetik

Comment 33

[Matthew Gregan [:kinetik]]

Cleaned up version up for review: https://github.com/kinetiknz/cubeb/pull/434

Try push with updated patch and some tests queued: https://treeherder.mozilla.org/#/jobs?repo=try&revision=86cef0d4a2d7cdb508b5b7a326e57510842e9c5f

Comment 34

[Matthew Gregan [:kinetik]]

Thanks for the testing and feedback, asesg4 and samer!  I'll post a link to the "final" test build once it completes.

Comment 35

[Matthew Gregan [:kinetik]]

Try push is actually: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6732f8777511c0b5bece2dcb9908980f6c4da84c

Previous attempt failed because trychooser generates try instructions that the decision task rejects. :-/

Comment 36

[Matthew Gregan [:kinetik]]

Latest test build: https://queue.taskcluster.net/v1/task/E2IlrZO4Qw6qN-caa6Qn8Q/runs/0/artifacts/public/build/target.dmg

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5cf87a14afb5
Apply latency correction to audio stream position on macOS. r=padenot

Comment 38

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/5cf87a14afb5