1749539 - (VideoFrame) [meta] Implement VideoFrame

Status: NEW

(Core :: Audio/Video: Web Codecs, enhancement, P1)

Description

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Implement the VideoFrame interface.

VideoFrames will be produced in the platform by:

1. The MediaStreamTrackProcessor API (bug 1749532)
2. The VideoDecoder (bug 1749045)
3. The VideoFrame constructor, from any CanvasImageSource (e.g. Image) or even a BufferSource (e.g. ArrayBuffer).

Given 3, it might be possible to hand this over to Graphics to untangle tasks, but I'm filing it here for now since it's technically part of WebCodecs.

Comment 1

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1749539 - Add a blank VideoColorSpace interface

This patch creates a blank class for the VideoColorSpace interface. The
files are generated by running ./mach build-backend && ./mach webidl-example VideoColorSpace with necessary changes to make it
buildable.

The implementations are plain blank now. They will be filled out in the
following patches.

Comment 2

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1749539 - Add a blank VideoFrame interface

This patch creates a blank class for the VideoFrame interface. The files
are generated by running ./mach build-backend && ./mach webidl-example VideoFrame with necessary changes to make it buildable.

The implementations are plain blank now. They will be filled out in the
following patches.

Depends on D144771

Comment 3

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1749539 - Implement VideoColorSpace

Depends on D144772

Comment 4

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1749539 - Implement basic VideoFrame interfaces

Comment 5

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1749539 - Expose some helpers from ImageBitmap

Depends on D145395

Comment 6

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1749539 - Implement VideoFrame Constructor for I420 ArrayBuffer*

Depends on D146752

Comment 7

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1749539 - Implement VideoFrame Constructor for ImageBitmap

Depends on D148082

Comment 8

[C.M.Chang[:chunmin]]

I am going to divide this bug into several smaller sub-tasks, and make this a meta bug for VideoFrame. After working on VideoFrame for a while, I realize that we need to add and/or modify some internal gfx data structure to support all the formats defined in the spec. Landing some basic operations first allows us to unblock other webcodec's works.

Comment 9

[Phabricator Automation]

Comment on attachment 9273949 [details]
WIP: Bug 1749539 - Add a blank VideoColorSpace interface

Revision D144771 was moved to bug 1774300. Setting attachment 9273949 [details] to obsolete.

Comment 10

[Phabricator Automation]

Comment on attachment 9273950 [details]
WIP: Bug 1749539 - Add a blank VideoFrame interface

Revision D144772 was moved to bug 1774300. Setting attachment 9273950 [details] to obsolete.

Comment 11

[Phabricator Automation]

Comment on attachment 9274909 [details]
WIP: Bug 1749539 - Implement VideoColorSpace

Revision D145395 was moved to bug 1774300. Setting attachment 9274909 [details] to obsolete.

Comment 12

[Phabricator Automation]

Comment on attachment 9277213 [details]
WIP: Bug 1749539 - Implement basic VideoFrame interfaces

Revision D146752 was moved to bug 1774300. Setting attachment 9277213 [details] to obsolete.

Comment 13

[Phabricator Automation]

Comment on attachment 9279353 [details]
WIP: Bug 1749539 - Implement VideoFrame Constructor for I420 ArrayBuffer*

Revision D148082 was moved to bug 1774300. Setting attachment 9279353 [details] to obsolete.

Comment 14

[Phabricator Automation]

Comment on attachment 9280705 [details]
WIP: Bug 1749539 - Implement VideoFrame Constructor for ImageBitmap

Revision D148930 was moved to bug 1774300. Setting attachment 9280705 [details] to obsolete.

Comment 15

[Marc]

Is there any progress on this?
I don't know how I should tell people to still use Firefox if background blur in video meetings are not working because this feature is missing?
Other browsers implemented this 2 years ago. Even Safari has it since 6 months.

https://www.3cx.com/community/threads/background-blur-for-3cx-video-meetings.122356/
https://developer.mozilla.org/en-US/docs/Web/API/VideoFrame#browser_compatibility