1902115 - Implement convert-to-RGB-frame for VideoFrame copyto method

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, task, P1)

Description

[C.M.Chang[:chunmin]]

Currently most WPTs in videoFrame-copyTo-rgb fail. We make it work unless some underlying image formats are unsupported.

---

update: The display-p3 color conversion will be fixed in bug 1904471 instead.

Comment 1

[C.M.Chang[:chunmin]]

VideoFrameCopyToOptions has to be updated to meet the latest spec:

dictionary VideoFrameCopyToOptions {
  DOMRectInit rect;
  sequence<PlaneLayout> layout;
  VideoPixelFormat format;
  PredefinedColorSpace colorSpace;
};

The spec was changed in https://github.com/w3c/webcodecs/pull/754

Comment 2

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Update ParseVideoFrameCopyToOptions error type to MediaResult

This patch updates the error type in ParseVideoFrameCopyToOptions to
MediaResult. A subsequent patch will introduce the use of
NotSupportedError for unsupported formats instead of TypeError. This
change helps clarify the conditions under which NotSupportedError will
be employed.

Comment 3

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Update VideoFrameCopyToOptions webidl

This patch updates the VideoFrameCopyToOptions webidl to meet the
changes in WebCodecs PR 754 [1].

[1] https://github.com/w3c/webcodecs/pull/754

Depends on D213804

Comment 4

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Update ParseVideoFrameCopyToOptions

This patch updates the ParseVideoFrameCopyToOptions to the changes
made in WebCodecs PR 754 [1].

[1] https://github.com/w3c/webcodecs/pull/754

Depends on D213805

Comment 5

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Alphabetize forward declarations in VideoFrame.cpp

Depends on D213806

Comment 6

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Implement convert-to-RGB-frame for copyto method

This patch implements the convert-to-RGB-frame algorithm [1] and
introduces it to copyto method to meet the latest spec [2].

[1] https://w3c.github.io/webcodecs/#videoframe-convert-to-rgb-frame
[2] https://w3c.github.io/webcodecs/#dom-videoframe-copyto

Depends on D214199

Comment 7

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Implement clone-configuration for VideoFrameCopyToOptions

This patch implements a helper function for copyto method,
clone-configuration algorithm for VideoFrameCopyToOptions.

Depends on D214200

Comment 8

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Implement ConvertToColorSpace

Depends on D214201

Comment 9

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Implement ConvertToRGBAImage

Depends on D214202

Comment 10

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Alphabetize include headers in ImageConversion.cpp

Depends on D214203

Comment 11

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Implement ConvertToRGBA

Depends on D214440

Comment 12

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Update WPT expectations

Depends on D214441

Comment 13

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - No conversion needed if colors and formats are same

Depends on D214473

Comment 14

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Refactor VideoFrame.copyTo Tests by colorspace

This patch refactors the tests in videoFrame-copyTo-rgb.any.js by
separating them according to the color space specified in the
VideoFrame. Previously, VideoFrame instances with different color spaces
were combined into a single test. Now, each color space specified in a
VideoFrame has its own dedicated test. This division facilitates easier
identification of missing color conversion and provides cleared insights
into completed and pending work for the following patches.

Comment 15

[C.M.Chang[:chunmin]]

Attachment: Bug 1902115 - Remove unnecessary I420 VideoFrame creation

In makeI420Frames, there is a VideoFrame created without meaningful
YUV data. This redundant creation should be removed.

Depends on D214882

Comment 16

[C.M.Chang[:chunmin]]

Attachment: videoFrame-copyTo-rgb.html

This is a visualized test page I used to debug during the development.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5083bce859c9
Refactor VideoFrame.copyTo Tests by colorspace r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/cb18d1d362be
Update ParseVideoFrameCopyToOptions error type to MediaResult r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/e789b595dd0f
Update VideoFrameCopyToOptions webidl r=webidl,media-playback-reviewers,smaug,padenot
https://hg.mozilla.org/integration/autoland/rev/9fe5e83ee26c
Update ParseVideoFrameCopyToOptions r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/97ae0a4994b5
Alphabetize forward declarations in VideoFrame.cpp r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/8e23738fdfd5
Implement convert-to-RGB-frame for copyto method r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/95b839237bae
Implement clone-configuration for VideoFrameCopyToOptions r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/6aed2000e65f
Implement ConvertToColorSpace r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/085ecd57565b
Implement ConvertToRGBAImage r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/f20aa24e0334
Alphabetize include headers in ImageConversion.cpp r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/8388eea038fe
Implement ConvertToRGBA r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/e4f215f2955b
Update WPT expectations r=padenot
https://hg.mozilla.org/integration/autoland/rev/22d5e85f3c3b
No conversion needed if colors and formats are same r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/2da61495720a
Remove unnecessary I420 VideoFrame creation r=media-playback-reviewers,padenot

Comment 18

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/47068 for changes under testing/web-platform/tests

Comment 19

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/5083bce859c9
https://hg.mozilla.org/mozilla-central/rev/cb18d1d362be
https://hg.mozilla.org/mozilla-central/rev/e789b595dd0f
https://hg.mozilla.org/mozilla-central/rev/9fe5e83ee26c
https://hg.mozilla.org/mozilla-central/rev/97ae0a4994b5
https://hg.mozilla.org/mozilla-central/rev/8e23738fdfd5
https://hg.mozilla.org/mozilla-central/rev/95b839237bae
https://hg.mozilla.org/mozilla-central/rev/6aed2000e65f
https://hg.mozilla.org/mozilla-central/rev/085ecd57565b
https://hg.mozilla.org/mozilla-central/rev/f20aa24e0334
https://hg.mozilla.org/mozilla-central/rev/8388eea038fe
https://hg.mozilla.org/mozilla-central/rev/e4f215f2955b
https://hg.mozilla.org/mozilla-central/rev/22d5e85f3c3b
https://hg.mozilla.org/mozilla-central/rev/2da61495720a

Comment 20

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot