Closed Bug 776137 Opened 9 years ago Closed 3 years ago

Video and Sound (Youtube) asynchronous if AirPlay is enabled

Categories

(Core :: Audio/Video: cubeb, defect, P2)

x86
macOS
defect

Tracking

()

RESOLVED FIXED
mozilla61
Tracking Status
platform-rel --- +
firefox61 --- fixed

People

(Reporter: Nomis101, Assigned: kinetik)

References

(Blocks 1 open bug)

Details

(Whiteboard: [platform-rel-Youtube])

If you enable AirPlay on Mountain Lion, than the sound has a delay of a few seconds (I think because of error correction). Apple has solved this for Quicktime and Quicklook by also delaying the video the same time. This also works in Safari 6 if you watch Youtube videos. So the sound and the video is synchronous again if AirPlay is enabled (works for Youtube, but not for other video sites). If you watch Youtube videos in Firefox, than this doesn't work and Video and Sound is asynchronous.

10.8 12A269
Thanks for the report.  We can fix this for HTML 5 video if that's where the issue lies, but Flash's audio is out of our control and will require an update from Adobe if you're seeing this problem with Flash video playback.

Anyway, this isn't too surprising for our side; the way we calculate the audio playback position depends on the audio callback running frequently with low latency (see http://mxr.mozilla.org/mozilla-central/source/media/libsydneyaudio/src/sydney_audio_mac.c#431).  We're using that technique to work around issues we had with CoreAudio's clock reporting bad values on suspend/resume and device changes.

Fixing this correctly probably means moving back to using CoreAudio's clock, which'll mean we need to find another way to address the other issues.
Exactly, sorry I haven't mentioned this. In Safari I use the YouTube5 extension, so the videos in Youtube are all HTML5. If I switch to Flash, than it's also asynchronous in Safari. In Firefox both, HTML5 and Flash, is asynchronous.
Component: Audio/Video → Audio/Video: Playback
Fixing this would be appreciated. It's annoying to have to use another browser for youtube links :)
(In reply to contact from comment #3)
> Fixing this would be appreciated. It's annoying to have to use another
> browser for youtube links :)

Is this issue still reproducible? Is the audio early or late?
(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #4)
> Is this issue still reproducible? Is the audio early or late?

If reproducible (I can't check at the moment, not having an AirPlay device nearby), the sound was late. Basically it seemed like Firefox did not do what The Other Browser does here - it holds the video playback for a few seconds to sync it with the output audio buffer of the audio device. In The Other Browser it works with video only (not interactive content), which would support the observation that it's about delaying the video rendering to sync with audio. 
In Firefox, last time I checked, simply audio and video start their playback together, but when streaming through AirPlay, audio gets a delay.
(In reply to contact from comment #5)
> In Firefox, last time I checked, simply audio and video start their playback
> together, but when streaming through AirPlay, audio gets a delay.

I checked just know with latest Nightly. And this is still the case. Over Airplay the sound is coming a few seconds too late.
The stream clock in cubeb_audiounit.c (audiounit_stream_get_position) doesn't adjust for hardware or playout latency.  We have the code (added much later) in audiostream_stream_get_latency to query the hardware latency and the playout latency looks trivial to calculate, so it would be fairly simple to fix this.
Assignee: nobody → kinetik
platform-rel: --- → ?
Whiteboard: [platform-rel-Youtube]
platform-rel: ? → +
Blake - do you think this would be a good fit for Chun Min after he's finished with 5.1 stuff?
Flags: needinfo?(bwu)
(In reply to Matthew Gregan [:kinetik] from comment #7)
> The stream clock in cubeb_audiounit.c (audiounit_stream_get_position)
> doesn't adjust for hardware or playout latency.  We have the code (added
> much later) in audiostream_stream_get_latency to query the hardware latency
> and the playout latency looks trivial to calculate, so it would be fairly
> simple to fix this.

It turns out this doesn't work as assumed, so there wasn't a quick fix for this.  The playout latency reported by audiostream_stream_get_latency doesn't seem to include delay induced by AirPlay (although it should be correctly queried via the device_latency_frames request, possibly a bug there).  Looking around at the time, it looked like the solution was to register notifications to detect when the audio device switched to and from an AirPlay device (which we now have most/all of the code for) and then add a fixed 2000ms offset to the clock to repair A/V sync.

Some history of VLC trying to fix this: https://trac.videolan.org/vlc/ticket/7127
And eventually reverting the changes due to problems: https://trac.videolan.org/vlc/ticket/14042
I can see this problem on my macOS Sierra but audio comes earlier than video. FWIW, Chrome doesn't have this problem. 

It is an very interesting bug. Yeah, it is a good bug for Chunmin to check and know more about cubeb. He should be fascinated to this. 

Matthew,
Do you mind letting Chunmin to check this bug?
Flags: needinfo?(bwu)
Summary: [10.8] Video and Sound (Youtube) asynchronous if AirPlay is enabled → Video and Sound (Youtube) asynchronous if AirPlay is enabled
Assignee: kinetik → cchang
I tested with AppleTV 3 in the meeting room but it works fine. The audio and video are synchronous. Blake will help us to test with his Apple TV 2 and see what could we find.
I tried the latest nightly with my Apple TV 3 again. And now I cannot see AV unsync problem. It looks like something magically changed on the nightly and it fixed this problem somehow. 

ChunMin, 
BTW, I was wrong about the version[1] of Apple TV in our office. It should be Apple TV "4".  

[1]https://support.apple.com/en-us/HT200008
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
I still have a 2 second latency between audio and video when connected to a speaker via AirPlay.
I'm running on OSX 10.12.3, tested on both FF 56.0a1 and 54.0.1
Can you please re-open this bug?
(In reply to samer from comment #13)
> I still have a 2 second latency between audio and video when connected to a
> speaker via AirPlay.
> I'm running on OSX 10.12.3, tested on both FF 56.0a1 and 54.0.1
> Can you please re-open this bug?

Thanks for the report.  Which device and version are you using at the output side, an Apple TV (which version?) or something else?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Sorry, I should have mentioned that I'm using a Raspberry Pi running Shairport-Sync[0]. I have previously filed a bug report[1] to the maintainer, but he claims that the issue is particular to Firefox:

> There is always some delay when you send audio to an Airplay device such as Shairport Sync or the Apple Airport Express. The delay is set by the audio sender and is in the range of about 400 milliseconds to over two seconds. Most senders set the delay at two seconds, including the Mac, when you're routing system audio to an AirPlay device. To compensate for the delay, Safari and Chrome and the Quicktime Player delay video part of a movie by two seconds so that the video and audio are in sync. Firefox doesn't do it. Also, Safari and Chrome don't do it for all YouTube content.\

I can confirm that I am only facing the problem with Firefox, not Safari or Chrome.

[0] https://github.com/mikebrady/shairport-sync/
[1] https://github.com/mikebrady/shairport-sync/issues/161)
I have the same issue. Playing a youtube video with Firefox results in audio and video being out of sync. Chrome and Safari play the same youtube video just fine with audio matching the video.

I'm on a MacBook Pro 5,3 with OS X 10.11.6 and Firefox 54.0.1 (64-bit). I use an Apple Airport Express (1st Generation, v7.6.8) as my Wi-Fi router and to stream audio via Airplay.

Steps to reproduce:

1. Connect to Airport Express
2. Set the Audio Output Device to the Airplay device
3. Open a youtube video

Expected result: Video plays with audio and video in sync.
Actual result: Video plays, but audio lags approximately 2 seconds behind.
Mass change P1->P2 to align with new Mozilla triage process
Priority: P1 → P2
Hi Reporters, 
Can you help test with Firefox 57 to see if this is still a problem or not?
Thanks.
(In reply to Blake Wu [:bwu][:blakewu] from comment #18)
> Hi Reporters, 
> Can you help test with Firefox 57 to see if this is still a problem or not?
> Thanks.

Unfortunately I have no access to my Airport Express at the moment, so I cannot test.
The problem persists for me when running Firefox 57 beta.
MacOS 10.12 + Raspberry Pi running Shairport-sync.
(In reply to samer from comment #20)
> The problem persists for me when running Firefox 57 beta.
> MacOS 10.12 + Raspberry Pi running Shairport-sync.
I cannot repro this bug with the Firefox 57.0b11 and the latest nightly 58 on macOS 10.13 (17A405) with Apple TV (3rd generation). 

What is your Apple TV version? 
https://support.apple.com/en-us/HT200008
(In reply to Blake Wu [:bwu][:blakewu] from comment #21)
> (In reply to samer from comment #20)
> > The problem persists for me when running Firefox 57 beta.
> > MacOS 10.12 + Raspberry Pi running Shairport-sync.
> I cannot repro this bug with the Firefox 57.0b11 and the latest nightly 58
> on macOS 10.13 (17A405) with Apple TV (3rd generation). 
> 
> What is your Apple TV version? 
> https://support.apple.com/en-us/HT200008

As mentioned above, I'm emulating an Apple TV using a Raspberry Pi and Shairport-sync (https://github.com/mikebrady/shairport-sync)
Problem still persists with Firefox 56 and Denon AVR X2000 receiver.
If now AV unsync issue only happens on Shairport-sync, not Apple TV, it would be better to file another bug and close this one.
(In reply to Blake Wu [:bwu][:blakewu] from comment #24)
> If now AV unsync issue only happens on Shairport-sync, not Apple TV, it
> would be better to file another bug and close this one.

As I mentioned, not only on Shairport-sync, but also on AirPlay compatible amplifiers.
Can confirm also experiencing audio latency when using airplay.

Firefox: 58.0b7 (64-bit)
MacOS 10.13.1
Airplay Audio Device: Sony Speaker RDP-XA900iP

Latency does not exist when using Safari (Version 11.0.1 (13604.3.5)) or other applications like iTunes.
The problem persists with Firefox 59.0.1 on macOS 10.13.3. There extists a fixed 2 second audio delay when airplaying to Apple HomePod. The problem doesn't affect Safari or Chrome.

For your reference this is the chromium bug regarding this problem: https://bugs.chromium.org/p/chromium/issues/detail?id=783282
(In reply to asesg4 from comment #27)
> For your reference this is the chromium bug regarding this problem:
> https://bugs.chromium.org/p/chromium/issues/detail?id=783282

Thanks, that's useful.  We've got code that queries kAudioDevicePropertyLatency for device enumeration, but we're not using it for latency adjustment.  From a quick test with my ATV4, I get surprisingly low values (~150ms)... I'll experiment a bit more and see if I can get it working correctly.
Assignee: chun.m.chang → kinetik
Component: Audio/Video: Playback → Audio/Video: cubeb
Nightly test build here: https://index.taskcluster.net/v1/task/gecko.v2.try.revision.d7512296fab96bae8ab91979bca41ae8b892db62.firefox.macosx64-opt/artifacts/public/build/target.dmg

This looks and sounds right for local playback and using AirPlay to an Apple TV 4 (which is reporting ~150ms).  I'd appreciate feedback on how well this works for others, particular anyone with device that has a ~2s delay.
NI to myself so I test with my Bluetooth setup at home that has high latency.
Flags: needinfo?(padenot)
The issue is fixed in the test build. Audio and video now syncs on my HomePod. Thanks.
Nightly works for me!
Emulating airplay using a Raspberry Pi with Shairport-Sync.

Thank you so much :kinetik
Thanks for the testing and feedback, asesg4 and samer!  I'll post a link to the "final" test build once it completes.
Try push is actually: https://treeherder.mozilla.org/#/jobs?repo=try&revision=6732f8777511c0b5bece2dcb9908980f6c4da84c

Previous attempt failed because trychooser generates try instructions that the decision task rejects. :-/
Pushed by mgregan@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5cf87a14afb5
Apply latency correction to audio stream position on macOS. r=padenot
https://hg.mozilla.org/mozilla-central/rev/5cf87a14afb5
Status: REOPENED → RESOLVED
Closed: 4 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
Flags: needinfo?(padenot)
You need to log in before you can comment on or make changes to this bug.