Closed Bug 812650 Opened 7 years ago Closed 7 years ago

android webdev wants to capture audio using mozGetUserMedia

Categories

(Core :: WebRTC, defect, P2)

ARM
Android
defect

Tracking

()

VERIFIED FIXED
mozilla21
Tracking Status
relnote-firefox --- -

People

(Reporter: dmose, Assigned: gcp)

References

(Blocks 1 open bug)

Details

(Whiteboard: [getUserMedia], [blocking-gum-])

Attachments

(1 file, 3 obsolete files)

The ultimate goal here is to be able to capture playable audio using mozGetUserMedia on Android.  A manual test that demonstrates this is the test shown by the Audio button on <http://mozilla.github.com/webrtc-landing/gum_test.html>.  If you have press that button with headphones on, you should hear yourself speak.

A potentially reasonable implementation strategy: import the patch from https://bugzilla.mozilla.org/show_bug.cgi?id=781534 into your mercurial queue.  Run the mochitests on your Android and ensure that they fail as expected.  Then 
code until dom/media/tests/mochitest/test_getUserMedia_basicAudio.html passes.
Making this work will likely require adding code to dynamically link against OpenSLES, at the very least.  The idea here is Android 2.2 (our current baseline) doesn't have that, and we want Fennec on Android to gracefully fail to load that library rather than crashing at startup, as it would do if we statically linked against it.
Priority: -- → P2
Hardware: x86 → ARM
Whiteboard: [getUserMedia], [blocking-gum-]
Turns out this gets us off to a pretty good start...
Assignee: nobody → gpascutto
Not sure what exactly is up yet. That getOutput() call seems suspect, but no idea yet if it's really that that is causing the test to bail.

I/GeckoDump( 7638): 15 INFO TEST-START | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudio.html
I/PRLog   ( 7638): 2628824[68082a50]: Selected audio device
I/PRLog   ( 7638): 1779808[5c651080]: Starting up MediaStreamGraph 61afe600
I/PRLog   ( 7638): 1779808[5c651080]: Returning success for getUserMedia()
I/PRLog   ( 7638): 2628824[68082a50]: started all sources
I/PRLog   ( 7638): 2628824[68082a50]: Selected audio device
I/PRLog   ( 7638): 4604232[68081f60]: Media graph global underrun detected
I/PRLog   ( 7638): 4604232[68081f60]: Advancing mStateComputedTime from 0.000001 to 0.003674
I/PRLog   ( 7638): 4604232[68081f60]: Updating current time to 0.003674 (real 0.003673, mStateComputedTime 0.003674)
I/PRLog   ( 7638): 4604232[68081f60]: Adding media stream 61de8740 to the graph
I/PRLog   ( 7638): 4604232[68081f60]: SourceMediaStream 61de8740 creating track 2, rate 16000, start 0, initial end 0
I/PRLog   ( 7638): 4604232[68081f60]: Media graph 61afe600 computing blocking for time 0.003674
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de8740 will block due to data underrun, bufferEnd 0.003674
I/PRLog   ( 7638): 4604232[68081f60]: Media graph 61afe600 computed blocking for interval 0.003674 to 0.033673
I/PRLog   ( 7638): 4604232[68081f60]: Waiting for next iteration; at 0.004252, timeout=0.010000
I/PRLog   ( 7638): 1779808[5c651080]: Returning success for getUserMedia()
I/PRLog   ( 7638): 2628824[68082a50]: started all sources
[...]
V/AudioPolicyManager(  240): getDeviceForStrategy() from cache strategy 0, device 2
V/AudioPolicyManager(  240): getOutput() stream 3, samplingRate 0, format 0, channels c, flags 0
I/Gecko - SYDNEY_AUDIO( 7638): 0x62165ac0 - New stream 16000 1 bsz=32000 min=2972 obsz=8192
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de8740 writing samples for 0.055095 to 0.065671 (samples 0 to 169)
I/PRLog   ( 7638): 4604232[68081f60]: Waiting for next iteration; at 0.039150, timeout=0.007000
I/PRLog   ( 7638): 4604232[68081f60]: Resuming after timeout; at 0.046288, elapsed=0.007142
I/PRLog   ( 7638): 4604232[68081f60]: Updating current time to 0.046366 (real 0.046367, mStateComputedTime 0.065671)
[...]
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de87f0 will block due to data underrun, bufferEnd 0.140767
I/PRLog   ( 7638): 4604232[68081f60]: Media graph 61afe600 computed blocking for interval 0.140767 to 0.151381
I/PRLog   ( 7638): 4604232[68081f60]: MediaStream 61de8740 writing blocking-silence samples for 0.140767 to 0.151381
I/PRLog   ( 7638): 4604232[68081f60]: Waiting for next iteration; at 0.121818, timeout=0.010000
I/GeckoDump( 7638): 16 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/media/tests/mochitest/test_peerConnection_basicAudio.html | Unexpected error callback with 1685598687
My expectation is that there are still enough unfixed problems in the signalling code on Android that I would try to enough find or tweak the tests you're developing against to ONLY do getUserMedia, not touch to PeerConnections at all.

Separately from that, if the above log was made without the patch from bug 819856 applied, you might want to give that a try and see if it helps at all.
There's no crash when running only the basicAudio test. When running the entire mochitest test dir, we crash in basicAudio with the following backtrace:
http://www.pastebin.mozilla.org/2038085
This is on a Galaxy Tab 10.1, which is a Tegra 2 device, which is ARMv7-without NEON. I'm waging a guess here that our runtime NEON detection is broken?
Could you try which tests have to be run in which order to cause basicAudio to crash?
I'm running the entire mochitest dir, so I think only test_getUserMedia_exceptions.html sits before it.
Can you please test this by removing individual tests from the dom/media/tests/mochitests/Makefile.in? I think it would be helpful to know. Also running a single file vs. against a whole folder can result in different behavior.
Attached patch patch 2, WIP (obsolete) — Splinter Review
Amazingly, with this slightly-tweaked WIP patch, I've seen (sometimes!)

* all the signaling tests pass
* all the mochitests pass
* gUM audio capture and replay work on <http://mozilla.github.com/webrtc-landing/gum_test.html>
* ekr's audio-only call test page <http://mozilla.github.com/webrtc-landing/pc_audio_only_test.html> declare success

I have yet to see <https://webrtc-demo.vcap.mozillalabs.com/mobile.html> work yet, but so far the obstacles there have been assertions in Fennec's graphics layer, which need to be filed if they're persisting.

In addition to minor C++ code cleanups, we'll at probably want to add a couple of <uses-feature android:required="false"> decls to the manifest.
Attachment #690250 - Attachment is obsolete: true
The successes I saw above were with various media.* preferences tweaked in about:config.
Attached patch patch 3, WIP (obsolete) — Splinter Review
Rebased against current mozilla-inbound.
Attachment #698185 - Attachment is obsolete: true
Depends on: 827359
Things that still need happen here with this patch:

* restructure the ifdefs so that the else clause just goes away and works on all platforms.

* remove the "true" from the if, and update usage instructions to require setting hidden permission disabled pref the way desktop did

* add any appropriate <uses-feature> clauses to the manifest XML
I don't think more is needed. Picture support is currently broken unrelated to this bug.
Attachment #698514 - Attachment is obsolete: true
Attachment #702353 - Flags: review?(dmose)
Comment on attachment 702353 [details] [diff] [review]
Patch 1. Enable webrtc audio on Android

Review of attachment 702353 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me; r=dmose.
Attachment #702353 - Flags: review?(dmose) → review+
Blocks: 830942
https://hg.mozilla.org/mozilla-central/rev/bb4d05e6185a
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla21
Keywords: verifyme
QA Contact: jsmith
QA Contact: nobody
Already covered by existing automation.

Also marking verified per existing MWC demo script testing has confirmed this to work as a proof of concept.
Status: RESOLVED → VERIFIED
Flags: in-testsuite+
Keywords: verifyme
Clearing the relnote flag, as this feature will not be available to users in Fx21.
You need to log in before you can comment on or make changes to this bug.