1509316 - MediaCodec decoder not usable on Android somehow.

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Jean-Yves Avenard [:jya]]

STR:

1: go to about:config and set media.navigator.mediadatadecoder_vpx_enabled to true
2: Go to a webrtc room like mozilla.appear.in/media
3: Join the same room from another machine.

What happens: The video of the other person stays black

what should happen: you see the other person

https://searchfox.org/mozilla-central/source/media/webrtc/signaling/src/media-conduit/WebrtcMediaDataDecoderCodec.cpp#126 decoding actually succeed, you get an image. However this image appear black.
Weither the image itself is black, or we're unable to pain its content I don't know.

but something isn't working. The image as the right dimensions, everything looks good on the debugger

This is tested on an Android 8 device Google Pixel 1.

If you reset media.navigator.mediadatadecoder_vpx_enabled to false (the default for now) then the image is fine.

Somehow, using a hardware based image doesn't work.

Comment 1

[John Lin [:jhlin][:jolin]]

On Android, VideoData::MarkSentToCompositor()[1] must be called right before the frame is given to compositor (as in VideoSink in the playback pipeline [2]).

Is there a proper place in WebRTC decoding pipeline to call this method? By calling it in WebrtcMediaDataDecoderCodec at [3], the remote video is shown and looks okay on my Pixel 3. But when there are multiple outputs in the same returned promise or another frame arrives before MSG processing it, some frames could be dropped.

[1] https://searchfox.org/mozilla-central/source/dom/media/MediaData.h#473
[2] https://searchfox.org/mozilla-central/source/dom/media/mediasink/VideoSink.cpp#384
[3] https://searchfox.org/mozilla-central/source/media/webrtc/signaling/src/media-conduit/WebrtcMediaDataDecoderCodec.cpp#134

Comment 2

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1509316 - p1: move composite listening out of VideoData/VideoSink. r=jya,mattwoodrow

On Android, decoded buffers need to be send back to MediaCodec in order to be
rendered and/or recycled. The current mechanism introduced in bug 1299068 only
works for playback(VideoData/VideoSink) but not WebRTC(VideoFrame/VideoOutput).
Move the callback to SurfaceTextureImage because VideoData and VideoFrame both
own that when using MediaCodec, and move the notification to VideoFrameContainer
for both VideoSink and VideoOutput pass frames there for compositing.

Comment 3

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1509316 - p2: enable MediaCodec for WebRTC on Android. r=jya

Depends on D45771

Comment 4

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1509316 - p3: transfer image ownership when constructing VideoData. r=jya

Depends on D45772

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b4bee18bb499
p1: move composite listening out of VideoData/VideoSink. r=jya,mattwoodrow
https://hg.mozilla.org/integration/autoland/rev/817f14af91f0
p2: enable MediaCodec for WebRTC on Android. r=jya
https://hg.mozilla.org/integration/autoland/rev/d64c35b9c211
p3: transfer image ownership when constructing VideoData. r=jya

Comment 6

[Dorel Luca [:dluca]]

Backed out 3 changesets (bug 1509316) for MDA failures on dom/media/tests/mochitest/test_peerConnection_captureStream_canvas_2d.html

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=268832181&repo=autoland&lineNumber=2970

Push with failues:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=d64c35b9c21149f01d57e1e2608900253fc57643

Backout:
https://hg.mozilla.org/integration/autoland/rev/6121d0104cd004af92b82874103ed7feb0cc2c6a

Comment 7

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6121d0104cd0
Backed out 3 changesets for MDA failures on dom/media/tests/mochitest/test_peerConnection_captureStream_canvas_2d.html

Comment 8

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1509316 - p4: disable HW codec in WebRTC tests. r?pehrsons

Canvas.drawImage() currently doesn't work for video when using MediaCodec
decoders (bug 1526207). Some tests use it to verify the contents of captured
stream and will fail if MediaDataDecoder is turned on. Flip the pref for these
tests for now until that bug is fixed.

Comment 9

[John Lin [:jhlin][:jolin]]

Attachment: Bug 1509316 - p4: disable HW codec in WebRTC tests. r?pehrsons

Depends on D45773

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/526d623222f0
p1: move composite listening out of VideoData/VideoSink. r=jya,mattwoodrow
https://hg.mozilla.org/integration/autoland/rev/ee77371cc6e6
p2: enable MediaCodec for WebRTC on Android. r=jya
https://hg.mozilla.org/integration/autoland/rev/4f83d4be23af
p3: transfer image ownership when constructing VideoData. r=jya
https://hg.mozilla.org/integration/autoland/rev/5976eb8e6be6
p4: disable HW codec in WebRTC tests. r=pehrsons,drno

Comment 11

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/526d623222f0
https://hg.mozilla.org/mozilla-central/rev/ee77371cc6e6
https://hg.mozilla.org/mozilla-central/rev/4f83d4be23af
https://hg.mozilla.org/mozilla-central/rev/5976eb8e6be6