812650 - android webdev wants to capture audio using mozGetUserMedia

Status: VERIFIED FIXED

(Core :: WebRTC, defect, P2)

Description

[Dan Mosedale (:dmosedale, :dmose)]

The ultimate goal here is to be able to capture playable audio using mozGetUserMedia on Android.  A manual test that demonstrates this is the test shown by the Audio button on <http://mozilla.github.com/webrtc-landing/gum_test.html>.  If you have press that button with headphones on, you should hear yourself speak.

A potentially reasonable implementation strategy: import the patch from https://bugzilla.mozilla.org/show_bug.cgi?id=781534 into your mercurial queue.  Run the mochitests on your Android and ensure that they fail as expected.  Then 
code until dom/media/tests/mochitest/test_getUserMedia_basicAudio.html passes.

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

Making this work will likely require adding code to dynamically link against OpenSLES, at the very least.  The idea here is Android 2.2 (our current baseline) doesn't have that, and we want Fennec on Android to gracefully fail to load that library rather than crashing at startup, as it would do if we statically linked against it.

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: patch 1, WIP: just try using the cross-platform code

Turns out this gets us off to a pretty good start...

Comment 3

[Gian-Carlo Pascutto [:gcp]]

Not sure what exactly is up yet. That getOutput() call seems suspect, but no idea yet if it's really that that is causing the test to bail.

I/GeckoDump( 7638): 15 INFO TEST-START | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudio.html
I/PRLog   ( 7638): 2628824[68082a50]: Selected audio device
I/PRLog   ( 7638): 1779808[5c651080]: Starting up MediaStreamGraph 61afe600
I/PRLog   ( 7638): 1779808[5c651080]: Returning success for getUserMedia()
I/PRLog   ( 7638): 2628824[68082a50]: started all sources
I/PRLog   ( 7638): 2628824[68082a50]: Selected audio device
I/PRLog   ( 7638): 4604232[68081f60]: Media graph global underrun detected
I/PRLog   ( 7638): 4604232[68081f60]: Advancing mStateComputedTime from 0.000001 to 0.003674
I/PRLog   ( 7638): 4604232[68081f60]: Updating current time to 0.003674 (real 0.003673, mStateComputedTime 0.003674)
I/PRLog   ( 7638): 4604232[68081f60]: Adding media stream 61de8740 to the graph
I/PRLog   ( 7638): 4604232[68081f60]: SourceMediaStream 61de8740 creating track 2, rate 16000, start 0, initial end 0
I/PRLog   ( 7638): 4604232[68081f60]: Media graph 61afe600 computing blocking for time 0.003674
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de8740 will block due to data underrun, bufferEnd 0.003674
I/PRLog   ( 7638): 4604232[68081f60]: Media graph 61afe600 computed blocking for interval 0.003674 to 0.033673
I/PRLog   ( 7638): 4604232[68081f60]: Waiting for next iteration; at 0.004252, timeout=0.010000
I/PRLog   ( 7638): 1779808[5c651080]: Returning success for getUserMedia()
I/PRLog   ( 7638): 2628824[68082a50]: started all sources
[...]
V/AudioPolicyManager(  240): getDeviceForStrategy() from cache strategy 0, device 2
V/AudioPolicyManager(  240): getOutput() stream 3, samplingRate 0, format 0, channels c, flags 0
I/Gecko - SYDNEY_AUDIO( 7638): 0x62165ac0 - New stream 16000 1 bsz=32000 min=2972 obsz=8192
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de8740 writing samples for 0.055095 to 0.065671 (samples 0 to 169)
I/PRLog   ( 7638): 4604232[68081f60]: Waiting for next iteration; at 0.039150, timeout=0.007000
I/PRLog   ( 7638): 4604232[68081f60]: Resuming after timeout; at 0.046288, elapsed=0.007142
I/PRLog   ( 7638): 4604232[68081f60]: Updating current time to 0.046366 (real 0.046367, mStateComputedTime 0.065671)
[...]
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de87f0 will block due to data underrun, bufferEnd 0.140767
I/PRLog   ( 7638): 4604232[68081f60]: Media graph 61afe600 computed blocking for interval 0.140767 to 0.151381
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de8740 writing blocking-silence samples for 0.140767 to 0.151381
I/PRLog   ( 7638): 4604232[68081f60]: Waiting for next iteration; at 0.121818, timeout=0.010000
I/GeckoDump( 7638): 16 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudio.html | Unexpected error callback with 1685598687

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

My expectation is that there are still enough unfixed problems in the signalling code on Android that I would try to enough find or tweak the tests you're developing against to ONLY do getUserMedia, not touch to PeerConnections at all.

Separately from that, if the above log was made without the patch from bug 819856 applied, you might want to give that a try and see if it helps at all.

Comment 5

[Gian-Carlo Pascutto [:gcp]]

There's no crash when running only the basicAudio test. When running the entire mochitest test dir, we crash in basicAudio with the following backtrace:
http://www.pastebin.mozilla.org/2038085

Comment 6

[Gian-Carlo Pascutto [:gcp]]

This is on a Galaxy Tab 10.1, which is a Tegra 2 device, which is ARMv7-without NEON. I'm waging a guess here that our runtime NEON detection is broken?

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Could you try which tests have to be run in which order to cause basicAudio to crash?

Comment 8

[Gian-Carlo Pascutto [:gcp]]

I'm running the entire mochitest dir, so I think only test_getUserMedia_exceptions.html sits before it.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Can you please test this by removing individual tests from the dom/media/tests/mochitests/Makefile.in? I think it would be helpful to know. Also running a single file vs. against a whole folder can result in different behavior.

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: patch 2, WIP

Amazingly, with this slightly-tweaked WIP patch, I've seen (sometimes!)

* all the signaling tests pass
* all the mochitests pass
* gUM audio capture and replay work on <http://mozilla.github.com/webrtc-landing/gum_test.html>
* ekr's audio-only call test page <http://mozilla.github.com/webrtc-landing/pc_audio_only_test.html> declare success

I have yet to see <https://webrtc-demo.vcap.mozillalabs.com/mobile.html> work yet, but so far the obstacles there have been assertions in Fennec's graphics layer, which need to be filed if they're persisting.

In addition to minor C++ code cleanups, we'll at probably want to add a couple of <uses-feature android:required="false"> decls to the manifest.

Comment 11

[Dan Mosedale (:dmosedale, :dmose)]

The successes I saw above were with various media.* preferences tweaked in about:config.

Comment 12

[Dan Mosedale (:dmosedale, :dmose)]

Attachment: patch 3, WIP

Rebased against current mozilla-inbound.

Comment 13

[Dan Mosedale (:dmosedale, :dmose)]

Things that still need happen here with this patch:

* restructure the ifdefs so that the else clause just goes away and works on all platforms.

* remove the "true" from the if, and update usage instructions to require setting hidden permission disabled pref the way desktop did

* add any appropriate <uses-feature> clauses to the manifest XML

Comment 14

[Gian-Carlo Pascutto [:gcp]]

Attachment: Patch 1. Enable webrtc audio on Android

I don't think more is needed. Picture support is currently broken unrelated to this bug.

Comment 15

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 702353 [details] [diff] [review]
Patch 1. Enable webrtc audio on Android

Review of attachment 702353 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me; r=dmose.

Comment 16

[Gian-Carlo Pascutto [:gcp]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/bb4d05e6185a

Comment 17

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/bb4d05e6185a

Comment 18

[Jason Smith [:jsmith]]

Already covered by existing automation.

Also marking verified per existing MWC demo script testing has confirmed this to work as a proof of concept.

Comment 19

[bhavana bajaj [:bajaj]]

Clearing the relnote flag, as this feature will not be available to users in Fx21.