1868223 - Support [Serializable] for RTCEncodedVideoFrame and RTCEncodedAudioFrame (with .data member transfer)

Status: RESOLVED FIXED

(Core :: WebRTC: Networking, enhancement)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

This was changed in https://github.com/w3c/webrtc-encoded-transform/pull/182.

[Serializable] is needed to transfer RTCRtpScriptTransform's readable and writable streams elsewhere, e.g. to main-thread.

STRs:

1. Open https://jsfiddle.net/jib1/bf92opjh/
2. Click the Go! button (and give camera permission if needed)

Expected result:

• Two video elements playing, representing sender and receiver side.

Actual result:

• DataCloneError: RTCEncodedVideoFrame object could not be cloned.

Comment 1

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1868223: Add [Serializable] to RTCEncodedVideoFrame and RTCEncodedAudioFrame. r?bwc

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

There is an r+ patch which didn't land and no activity in this bug for 2 weeks.
:jib, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Comment 3

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Changing subject based on latest PR https://github.com/w3c/webrtc-encoded-transform/pull/238

Comment 4

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Latest PR: https://github.com/w3c/webrtc-encoded-transform/pull/239

Comment 5

[Denys [:denyshon]]

Jan-Ivar Bruaroey [:jib] (needinfo? me), could you please clarify the bug's status now? Is it fixed with the latest PR?

You also mentioned bug 1896361 being blocked by this bug. Could you please update the bug info to reflect that?

Comment 6

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

A second patch is needed to complete this fix. Trying to find time to work on it.

It's only blocking bug 1896361 insofar as building a reverse shim for facebook would require inefficient transfer of encoded frames between main thread and a worker, as explained in Bug 1896361 comment 13.

Bug 1896361 would ideally be fixed through other means: facebook updating to use RTCScriptTransform (part of Interop 2025) once Chrome turns it on. This would make it work in Safari as well, not just Firefox.

We can remove the block if that happens first.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1868223 - Add api:frame_transformer_factory to BUILD.gn r?mjf

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1868223 - Add moz.build files from updated BUILD.gn r?mjf

Comment 9

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1868223 - Implement custom RTCEncodedVideoFrame::Read/WriteStructuredClone for same process. r?bwc

Comment 10

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1868223 - Implement custom RTCEncodedAudioFrame::Read/WriteStructuredClone for same process. r?bwc

Comment 11

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Bug 1868223 - Reenable WPT RTCRtpScriptTransform-sender-worker-single-frame.https.html. r?bwc

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/84bd0063760c
https://hg.mozilla.org/integration/autoland/rev/08424d96ff02
Add api:frame_transformer_factory to BUILD.gn r=mjf
https://github.com/mozilla-firefox/firefox/commit/5c0daa9ed9ff
https://hg.mozilla.org/integration/autoland/rev/956e554be97a
Add moz.build files from updated BUILD.gn r=mjf
https://github.com/mozilla-firefox/firefox/commit/17b8e797b622
https://hg.mozilla.org/integration/autoland/rev/8b54d0607ebb
Implement custom RTCEncodedVideoFrame::Read/WriteStructuredClone for same process. r=bwc,webidl,smaug
https://github.com/mozilla-firefox/firefox/commit/e37d4c6dc5f5
https://hg.mozilla.org/integration/autoland/rev/bee55346f408
Implement custom RTCEncodedAudioFrame::Read/WriteStructuredClone for same process. r=bwc
https://github.com/mozilla-firefox/firefox/commit/7988aa915672
https://hg.mozilla.org/integration/autoland/rev/622978eb5067
Reenable WPT RTCRtpScriptTransform-sender-worker-single-frame.https.html. r=bwc

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/b8005febde95
https://hg.mozilla.org/integration/autoland/rev/0a340905a471
Revert "Bug 1868223 - Reenable WPT RTCRtpScriptTransform-sender-worker-single-frame.https.html. r=bwc" for causing ios build bustages @StructuredCloneHolder.cpp.

Comment 14

[agoloman]

Backed out for causing ios build bustages @StructuredCloneHolder.cpp.

• Backout link
• Push with failures
• Failure Log

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/b80009efc92d
https://hg.mozilla.org/integration/autoland/rev/8e7778c59894
Add api:frame_transformer_factory to BUILD.gn r=mjf
https://github.com/mozilla-firefox/firefox/commit/4fda8f0471f8
https://hg.mozilla.org/integration/autoland/rev/af35d7dd9df9
Add moz.build files from updated BUILD.gn r=mjf
https://github.com/mozilla-firefox/firefox/commit/4e31e678ec71
https://hg.mozilla.org/integration/autoland/rev/5cabce10e91c
Implement custom RTCEncodedVideoFrame::Read/WriteStructuredClone for same process. r=bwc,webidl,smaug
https://github.com/mozilla-firefox/firefox/commit/58a0249b524d
https://hg.mozilla.org/integration/autoland/rev/20f698043569
Implement custom RTCEncodedAudioFrame::Read/WriteStructuredClone for same process. r=bwc
https://github.com/mozilla-firefox/firefox/commit/35e4ff6e733e
https://hg.mozilla.org/integration/autoland/rev/c4e7bb567111
Reenable WPT RTCRtpScriptTransform-sender-worker-single-frame.https.html. r=bwc

Comment 16

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/8e7778c59894
https://hg.mozilla.org/mozilla-central/rev/af35d7dd9df9
https://hg.mozilla.org/mozilla-central/rev/5cabce10e91c
https://hg.mozilla.org/mozilla-central/rev/20f698043569
https://hg.mozilla.org/mozilla-central/rev/c4e7bb567111

Comment 17

[Hamish Willee]

FF145 MDN docs work for this done as part of https://github.com/mdn/content/issues/41508