Closed Bug 1509316 Opened 2 years ago Closed 10 months ago

MediaCodec decoder not usable on Android somehow.

Categories

(Core :: WebRTC: Audio/Video, defect, P2)

ARM
Android
defect

Tracking

()

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: jya, Assigned: jhlin)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 1 obsolete file)

STR:

1: go to about:config and set media.navigator.mediadatadecoder_vpx_enabled to true
2: Go to a webrtc room like mozilla.appear.in/media
3: Join the same room from another machine.

What happens: The video of the other person stays black

what should happen: you see the other person

https://searchfox.org/mozilla-central/source/media/webrtc/signaling/src/media-conduit/WebrtcMediaDataDecoderCodec.cpp#126 decoding actually succeed, you get an image. However this image appear black.
Weither the image itself is black, or we're unable to pain its content I don't know.

but something isn't working. The image as the right dimensions, everything looks good on the debugger

This is tested on an Android 8 device Google Pixel 1.

If you reset media.navigator.mediadatadecoder_vpx_enabled to false (the default for now) then the image is fine.

Somehow, using a hardware based image doesn't work.
Flags: needinfo?(jolin)
On Android, VideoData::MarkSentToCompositor()[1] must be called right before the frame is given to compositor (as in VideoSink in the playback pipeline [2]).

Is there a proper place in WebRTC decoding pipeline to call this method? By calling it in WebrtcMediaDataDecoderCodec at [3], the remote video is shown and looks okay on my Pixel 3. But when there are multiple outputs in the same returned promise or another frame arrives before MSG processing it, some frames could be dropped.

[1] https://searchfox.org/mozilla-central/source/dom/media/MediaData.h#473
[2] https://searchfox.org/mozilla-central/source/dom/media/mediasink/VideoSink.cpp#384
[3] https://searchfox.org/mozilla-central/source/media/webrtc/signaling/src/media-conduit/WebrtcMediaDataDecoderCodec.cpp#134
Flags: needinfo?(jolin)
Rank: 15
Priority: -- → P2
Assignee: nobody → jolin
Status: NEW → ASSIGNED

On Android, decoded buffers need to be send back to MediaCodec in order to be
rendered and/or recycled. The current mechanism introduced in bug 1299068 only
works for playback(VideoData/VideoSink) but not WebRTC(VideoFrame/VideoOutput).
Move the callback to SurfaceTextureImage because VideoData and VideoFrame both
own that when using MediaCodec, and move the notification to VideoFrameContainer
for both VideoSink and VideoOutput pass frames there for compositing.

Blocks: 1581175
Pushed by jolin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b4bee18bb499
p1: move composite listening out of VideoData/VideoSink. r=jya,mattwoodrow
https://hg.mozilla.org/integration/autoland/rev/817f14af91f0
p2: enable MediaCodec for WebRTC on Android. r=jya
https://hg.mozilla.org/integration/autoland/rev/d64c35b9c211
p3: transfer image ownership when constructing VideoData. r=jya
Backout by dluca@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6121d0104cd0
Backed out 3 changesets for MDA failures on dom/media/tests/mochitest/test_peerConnection_captureStream_canvas_2d.html

Canvas.drawImage() currently doesn't work for video when using MediaCodec
decoders (bug 1526207). Some tests use it to verify the contents of captured
stream and will fail if MediaDataDecoder is turned on. Flip the pref for these
tests for now until that bug is fixed.

Attachment #9092532 - Attachment description: Bug 1509316 - p1: move composite listening out of VideoData/VideoSink. r?jya → Bug 1509316 - p1: move composite listening out of VideoData/VideoSink. r=jya,mattwoodrow
Attachment #9092533 - Attachment description: Bug 1509316 - p2: enable MediaCodec for WebRTC on Android. r?jya → Bug 1509316 - p2: enable MediaCodec for WebRTC on Android. r=jya
Attachment #9092534 - Attachment description: Bug 1509316 - p3: transfer image ownership when constructing VideoData. r?jya → Bug 1509316 - p3: transfer image ownership when constructing VideoData. r=jya

Depends on D45773

Attachment #9098963 - Attachment is obsolete: true
Flags: needinfo?(jolin)
Pushed by jolin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/526d623222f0
p1: move composite listening out of VideoData/VideoSink. r=jya,mattwoodrow
https://hg.mozilla.org/integration/autoland/rev/ee77371cc6e6
p2: enable MediaCodec for WebRTC on Android. r=jya
https://hg.mozilla.org/integration/autoland/rev/4f83d4be23af
p3: transfer image ownership when constructing VideoData. r=jya
https://hg.mozilla.org/integration/autoland/rev/5976eb8e6be6
p4: disable HW codec in WebRTC tests. r=pehrsons,drno
You need to log in before you can comment on or make changes to this bug.