937718 - Silent output with cross-origin createMediaElementSource and Access-Control-Allow-Origin: *

Status: RESOLVED FIXED

(Core :: Web Audio, defect, P2)

Description

[Ian Kerr]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (Beta/Release)
Build ID: 20131112030204

Steps to reproduce:

Run the following JSFiddle, which uses Web Audio's createMediaElementSource() to route audio from an <audio> tag into a Web Audio context:

http://jsfiddle.net/ikerr/WcXHK/

To see debug output, open the Web Console (Tools > Web Developer > Web Console).  Note that the fiddle checks the audio context's sample rate and selects a matching 44100 Hz or 48000 Hz audio file (I haven't seen any other sample rates in my testing).

I tested this fiddle in Firefox 25, 27 (Aurora) and 28.0a1 (2013-11-12) on Mac OS X 10.9.  My system specs are as follows:

MacBook Pro, 15-inch, Mid 2012
Processor: 2.3 GHz Intel Core i7
Memory: 8 GB 1600 MHz DDR3
Graphics: NVIDIA GeForce GT 650M 512 MB


Actual results:

When this fiddle loads, I do not hear any music playing (i.e., silence).  When I run the fiddle, I get the following output in my Web Console (no errors):

10:02:28.005 "Context sample rate: 44100"
10:02:28.006 "Using audio asset: http://assets-staging.verold.com/entities/52812b86a9aff5020000000b/3f6b2c8c8996e748a753cf26c6f9406a/06+Old+Man-44100.mp3"
10:02:28.006 "Using audio asset: http://assets-staging.verold.com/entities/52812b86a9aff5020000000b/3f6b2c8c8996e748a753cf26c6f9406a/06+Old+Man-44100.ogg"
10:02:28.006 "Created MediaElementAudioSourceNode."


Expected results:

When this fiddle loads, music should start playing.  This works in the Chrome 30.0.1599.101, for example.  Documentation for createMediaElementSource() can be found here:

https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#MediaElementAudioSourceNode-section

Comment 1

[Karl Tomlinson (:karlt)]

Attachment: audio element testcase

This plays as expected.

Comment 2

[Karl Tomlinson (:karlt)]

Attachment: source element child of audio element testcase

This is reduced from http://jsfiddle.net/ikerr/WcXHK/
It doesn't produce sound.

Comment 3

[Karl Tomlinson (:karlt)]

Attachment: source element child of audio element testcase

Sorry.  This version, still without createMediaElementSource plays.

Comment 4

[Karl Tomlinson (:karlt)]

wget -S http://assets-staging.verold.com/entities/52812b86a9aff5020000000b/3f6b2c8c8996e748a753cf26c6f9406a/06+Old+Man-48000.ogg
indicates that "Access-Control-Allow-Origin: *" is returned.

Comment 5

[Karl Tomlinson (:karlt)]

Attachment: reduced testcase

Comment 6

[Karl Tomlinson (:karlt)]

Attachment: testcase with crossOrigin attribute without createMediaElementSource

Plays as expected.

Comment 7

[Karl Tomlinson (:karlt)]

Attachment: testcase with crossOrigin attribute and createMediaElementSource

No output.

Comment 8

[Azeem Arshad]

I am experience the same problem. jsfiddle that fails: http://jsfiddle.net/7bJUU/ 
In the above example, the ogg file from wikipedia is returned with "Access-Control-Allow-Origin: *" but the
output is silent. The same example works if 1) createMediaElementSource is not used and file is played directly
2) when a local, same origin file is played

Comment 9

[rbarber]

Additional test case: http://jsfiddle.net/xqhWY/2/

When the createMediaElementSource is created the output of the audio stops. The source is connected to the destination of the context, so I would expect output.

Comment 10

[rbarber]

I think I found the problem. -> http://jsfiddle.net/xqhWY/4/

The sample rate of the Audio Context is different then the sample rate of the audio stream. It appears in the Web Audio API spec that the AudioContext determines which sample rate to use <http://www.w3.org/TR/webaudio/#Sample-rate> and it assumes that other nodes will use the same rate <http://www.w3.org/TR/webaudio/#dfn-sampleRate>. This would explain the unexpected behaviour I'm seeing from Firefox when I attempt to create a media source from the audio element.

Comment 11

[Ian Kerr]

In the original bug report, I observed the problem even when the sample rate of the Audio Context matched the file's sample rate.  But I agree that when the sample rates do not match, the output should be silent.

(In reply to rbarber from comment #10)
> I think I found the problem. -> http://jsfiddle.net/xqhWY/4/
> 
> The sample rate of the Audio Context is different then the sample rate of
> the audio stream. It appears in the Web Audio API spec that the AudioContext
> determines which sample rate to use
> <http://www.w3.org/TR/webaudio/#Sample-rate> and it assumes that other nodes
> will use the same rate <http://www.w3.org/TR/webaudio/#dfn-sampleRate>. This
> would explain the unexpected behaviour I'm seeing from Firefox when I
> attempt to create a media source from the audio element.

Comment 13

[brian]

I can replicate this bug, with video elements as well.

However, it's not that the audio is silent, but rather that it doesn't play at all. Here's a modified version of the jsfiddle that displays the audio element with controls. You'll see that the timeline does not progress even though it is "playing" ... or at least, not paused.

http://jsfiddle.net/WcXHK/37/

Comment 14

[hooli]

sorry i dupe'd the bug again .. did not see this in my activity area till today :)

Comment 15

[hooli]

ps.. my context sample returns 'undefined'

Comment 17

[Paul Adenot (:padenot)]

Attachment: Output frames to the MediaElementAudioSourceNode when an <audio> has labeled the resource has CORS-cross-origin. r=

So, this is kind of what <canvas> does for drawImage with cross-origin I think.
I can see audio frames being appended to the MediaStream in
MediaDecoderStateMachine::SendStreamData, but then something else is broken and
it does not work, so a bit more work is needed, but I'd like to validate the
approach here.

Comment 18

[Paul Adenot (:padenot)]

Attachment: Output frames to the MediaElementAudioSourceNode when an <audio> has labeled the resource has CORS-cross-origin. r=

This is the rest of the work, it works, now. I'll r? when the mochitest is
ready, it kind of works but it too flaky.

Comment 19

[Paul Adenot (:padenot)]

https://tbpl.mozilla.org/?tree=Try&rev=63849cd43f96

Comment 20

[Paul Adenot (:padenot)]

Attachment: Output frames to the MediaElementAudioSourceNode when an <audio> has labeled the resource has CORS-cross-origin. r=

The silence was caused by the fact that the track was not enabled, because at
the MediaStream level, there was another origin check.

By teaching the existence of CORS to DOMMediaStream, and wiring that down to
MediaStreamAudioSourceNode::PrincipalChanged, we can track the cross-origin
status of the media that gets fed to the MSG, and mute/unmute accordingly.

Comment 21

[Paul Adenot (:padenot)]

Attachment: Test for MediaElementAudioSourceNode and cross-origin resources. r=

And the test, using an .sjs file to add the header. Green on try.

Comment 22

[Paul Adenot (:padenot)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0c9f97ebf7ea
https://hg.mozilla.org/integration/mozilla-inbound/rev/bcbc64d97fbf

Comment 23

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0c9f97ebf7ea
https://hg.mozilla.org/mozilla-central/rev/bcbc64d97fbf

Comment 25

[brian]

This still isn't working for me. I'm trying it on Nightly 38.0a1 (2015-02-09) on OSX 10.9.5. I tried in both an e10s and Non-e10s window with no success in either.

Can we reopen this?

Comment 26

[Paul Adenot (:padenot)]

(In reply to brian from comment #25)
> This still isn't working for me. I'm trying it on Nightly 38.0a1
> (2015-02-09) on OSX 10.9.5. I tried in both an e10s and Non-e10s window with
> no success in either.
> 
> Can we reopen this?

Open a new bug, please. Bonus points if you have a test case!

Comment 27

[WebAudioUser]

The example from bug 1111382 still doesn't work in the nightly: http://www.jlescure.com/samples/firefox-bug.html

Comment 28

[WebAudioUser]

(I was using 38.0a1 (2015-02-19) on OSX 10.10.1)

Comment 29

[WebAudioUser]

This example from further up in the bug does not work either: http://jsfiddle.net/7bJUU/

Comment 30

[dekztah]

i can confirm that this still wont work. even though soundcloud has access control allow origin on each of its mp3s, output is still silent in released firefox 37.0.1.

Comment 31

[dekztah]

update: it just skips a bit at the start, but will play if audio tag has crossorigin = 'anonymous' property!
disregard previous comment. in the jlescure.com example you should add audio.crossOrigin = 'anonymous'; and it also works :) thanks for fixing this!