1774300 - Implement basic VideoFrame

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, task, P1)

Description

[C.M.Chang[:chunmin]]

This bug is a sub-task for bug 1749539. The task is to implement the basic VideoFrame operations for some common formats.

The goal in this bug is to implement the VideoColorSpace interface and VideoFrame interface defined in the WebCodecs API, and make it pass the following tests:

• testing/web-platform/tests/webcodecs/videoFrame-construction.any.js
• testing/web-platform/tests/webcodecs/videoFrame-construction.window.js
• testing/web-platform/tests/webcodecs/videoFrame-copyTo.any.js

For now, we won't support all the Pixel Format since gecko only uses part of its defined formats. The supported formats in the implementation will be:

• I420
• I420A
• NV12
• RGBA
• RGBX
• BGRA
• BGRX

The I422 and I444 formats will be implemented in bug 1779655 and bug 1779656 respectively.

Aside from that, the Serializable and Transferable of the VideoFrame won't be implemented in this bug. They will be done in Bug 1774302 and Bug 1774306 respectively.

Comment 1

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Add a blank VideoColorSpace interface

This patch creates a blank class for the VideoColorSpace interface. The
files are generated by running ./mach build-backend && ./mach webidl-example VideoColorSpace with necessary changes to make it
buildable.

The implementations are plain blank now. They will be filled out in the
following patches.

Comment 2

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Add a blank VideoFrame interface

This patch creates a blank class for the VideoFrame interface. The files
are generated by running ./mach build-backend && ./mach webidl-example VideoFrame with necessary changes to make it buildable.

The implementations are plain blank now. They will be filled out in the
following patches.

Depends on D144771

Comment 3

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoColorSpace

Depends on D144772

Comment 4

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame interfaces for RGBA ArrayBuffer*

Implement the VideoFrame's Constructor for RGBA ArrayBuffer* and the
basic getters.

This patch pass the wpt test Test invalid buffer constructed VideoFrame by luck since constructing a VideoFrame from ArrayBuffer* is
acceptable now. However, we only support RGBA format in this patch so
that passed test doesn't be verified correctly. We will address this
in the following patches.

Depends on D145395

Comment 5

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for I420 ArrayBuffer*

Depends on D146752

Comment 6

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Remove unnecessary include

Including mozilla/layers/LayersSurfaces.h makes the next patch
unbuildable.

Depends on D148930

Comment 7

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for OffscreenCanvas

Depends on D149331

Comment 8

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for ImageBitmap

Comment 9

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for NV12 ArrayBuffer*

Depends on D148082

Comment 10

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for I420A ArrayBuffer*

Depends on D149584

Comment 11

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for VideoFrame

Depends on D149332

Comment 12

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for HTMLVideoElement

Depends on D150019

Comment 13

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Add a wpt test: construct a VideoFrame from a <video>

Depends on D150525

Comment 14

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Add VideoFrame into CanvasImageSource

Adding VideoFrame into CanvasImageSource so the spec is up to date
[1, 2].

[1] https://html.spec.whatwg.org/multipage/canvas.html#canvasimagesource
[2] https://github.com/w3c/webcodecs/issues/158

Depends on D144772

Comment 15

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for HTMLImageElement

Depends on D150526

Comment 16

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1774300 - Add a wpt test: construct a VideoFrame from a <img>

Depends on D150680

Comment 17

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Expose some SVG classes

Expose some SVG classes to mozilla/dom so VideoFrame can include
mozilla/dom/SVGImageElement.h and build gecko successfully.

Depends on D150681

Comment 18

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for SVGImageElement

Depends on D150836

Comment 19

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1774300 - Add a wpt test: construct a VideoFrame from a <svg> image

Depends on D150837

Comment 20

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Implement VideoFrame Constructor for HTMLCanvasElement

Depends on D150838

Comment 21

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1774300 - Add a wpt test: construct a VideoFrame from a <canvas>

Depends on D151091

Comment 22

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Add a wpt test: construct a VideoFrame with a smaller-than-codesize buffer

Depends on D151092

Comment 23

[Jeff Muizelaar [:jrmuizel]]

chunmin, can you write an overall summary of what's changing and why?

Comment 24

[C.M.Chang[:chunmin]]

Sure. I've updated comment 0 and all the summaries of the patches.

Comment 25

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Add VideoFrame into TexImageSource

This patch adds the VideoFrame into TexImageSource so the spec of
TexImageSource is up to date [1]

[1] https://registry.khronos.org/webgl/specs/latest/1.0/#5.14

Depends on D150679

Comment 26

[Phabricator Automation]

Comment on attachment 9287144 [details]
Bug 1774300 - Add VideoFrame into TexImageSource

Revision D152815 was moved to bug 1781526. Setting attachment 9287144 [details] to obsolete.

Comment 27

[Phabricator Automation]

Comment on attachment 9283547 [details]
Bug 1774300 - Add VideoFrame into CanvasImageSource

Revision D150679 was moved to bug 1781527. Setting attachment 9283547 [details] to obsolete.

Comment 28

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Add a wpt test: create VideoFrames from cross-origin CanvasImageSources

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4a145bdd6b86
Add a blank VideoColorSpace interface r=padenot,jgilbert,emilio
https://hg.mozilla.org/integration/autoland/rev/ba9bc5c631b0
Add a blank VideoFrame interface r=padenot,jgilbert,emilio
https://hg.mozilla.org/integration/autoland/rev/2bc06d0e8808
Implement VideoColorSpace r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/33b150b4c911
Implement VideoFrame interfaces for RGBA ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/50995e55c546
Implement VideoFrame Constructor for I420 ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/19f750d86a73
Implement VideoFrame Constructor for NV12 ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/825ae6e7be9b
Implement VideoFrame Constructor for I420A ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/e56596b91a86
Implement VideoFrame Constructor for ImageBitmap r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/4a7dca31703e
Remove unnecessary include r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/c5f67f22597e
Implement VideoFrame Constructor for OffscreenCanvas r=padenot
https://hg.mozilla.org/integration/autoland/rev/6ce76e0d6fd2
Implement VideoFrame Constructor for VideoFrame r=padenot
https://hg.mozilla.org/integration/autoland/rev/52fdd5a613f4
Implement VideoFrame Constructor for HTMLVideoElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/d5d263cedbc9
Add a wpt test: construct a VideoFrame from a <video> r=padenot
https://hg.mozilla.org/integration/autoland/rev/db37fffc8519
Implement VideoFrame Constructor for HTMLImageElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/56bfa6480cb5
Expose some SVG classes r=padenot
https://hg.mozilla.org/integration/autoland/rev/5f837d7e621b
Implement VideoFrame Constructor for SVGImageElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/3ab7aead37b5
Implement VideoFrame Constructor for HTMLCanvasElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/5af8771526a1
Add a wpt test: create VideoFrames from cross-origin CanvasImageSources r=padenot

Comment 30

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/36273 for changes under testing/web-platform/tests

Comment 31

[Atila Butkovits]

Backed out for causing lint failure.

Backout link: https://hg.mozilla.org/integration/autoland/rev/fd4addd084c6dff9b50fe4da66d7262858b5d323

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=f5460317d2440c225f4a6942453749168765f089&selectedTaskRun=fqk3yORcRFWxAc44SIRfAQ.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=392240661&repo=autoland&lineNumber=112

Comment 32

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 33

[C.M.Chang[:chunmin]]

Attachment: Bug 1774300 - Ignore no-undef for WebCodecs interfaces

VideoColorSpace and VideoFrame are defined behind a pref. Disable
the no-undef for now until the pref is enabled by default

Depends on D156022

Comment 34

[C.M.Chang[:chunmin]]

D158611 should be able to solve the lint issue.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c0cce5ee36e6
Add a blank VideoColorSpace interface r=padenot,jgilbert,emilio
https://hg.mozilla.org/integration/autoland/rev/543ebbcb3e78
Add a blank VideoFrame interface r=padenot,jgilbert,emilio
https://hg.mozilla.org/integration/autoland/rev/9301467e47ce
Implement VideoColorSpace r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/c77682ecaa83
Implement VideoFrame interfaces for RGBA ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/6ff9c84b02fc
Ignore no-undef for WebCodecs interfaces r=Standard8
https://hg.mozilla.org/integration/autoland/rev/1e65a9844adc
Implement VideoFrame Constructor for I420 ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/f347779c8963
Implement VideoFrame Constructor for NV12 ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/a1b280bfa6c1
Implement VideoFrame Constructor for I420A ArrayBuffer* r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/219fd6678030
Implement VideoFrame Constructor for ImageBitmap r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/bde8924b0c97
Remove unnecessary include r=padenot,jgilbert
https://hg.mozilla.org/integration/autoland/rev/cf5e4d9c1b91
Implement VideoFrame Constructor for OffscreenCanvas r=padenot
https://hg.mozilla.org/integration/autoland/rev/a692c13589cd
Implement VideoFrame Constructor for VideoFrame r=padenot
https://hg.mozilla.org/integration/autoland/rev/74a429c4333b
Implement VideoFrame Constructor for HTMLVideoElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/847b30b6f5bf
Add a wpt test: construct a VideoFrame from a <video> r=padenot
https://hg.mozilla.org/integration/autoland/rev/66761011e4d3
Implement VideoFrame Constructor for HTMLImageElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/61e59ea7d415
Expose some SVG classes r=padenot
https://hg.mozilla.org/integration/autoland/rev/adcf2d82738f
Implement VideoFrame Constructor for SVGImageElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/fa4c07395979
Implement VideoFrame Constructor for HTMLCanvasElement r=padenot
https://hg.mozilla.org/integration/autoland/rev/63d5939f41d3
Add a wpt test: create VideoFrames from cross-origin CanvasImageSources r=padenot

Comment 36

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/c0cce5ee36e6
https://hg.mozilla.org/mozilla-central/rev/543ebbcb3e78
https://hg.mozilla.org/mozilla-central/rev/9301467e47ce
https://hg.mozilla.org/mozilla-central/rev/c77682ecaa83
https://hg.mozilla.org/mozilla-central/rev/6ff9c84b02fc
https://hg.mozilla.org/mozilla-central/rev/1e65a9844adc
https://hg.mozilla.org/mozilla-central/rev/f347779c8963
https://hg.mozilla.org/mozilla-central/rev/a1b280bfa6c1
https://hg.mozilla.org/mozilla-central/rev/219fd6678030
https://hg.mozilla.org/mozilla-central/rev/bde8924b0c97
https://hg.mozilla.org/mozilla-central/rev/cf5e4d9c1b91
https://hg.mozilla.org/mozilla-central/rev/a692c13589cd
https://hg.mozilla.org/mozilla-central/rev/74a429c4333b
https://hg.mozilla.org/mozilla-central/rev/847b30b6f5bf
https://hg.mozilla.org/mozilla-central/rev/66761011e4d3
https://hg.mozilla.org/mozilla-central/rev/61e59ea7d415
https://hg.mozilla.org/mozilla-central/rev/adcf2d82738f
https://hg.mozilla.org/mozilla-central/rev/fa4c07395979
https://hg.mozilla.org/mozilla-central/rev/63d5939f41d3

Comment 37

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 38

[Alex Finder|PTO, returning August 18th]

(In reply to Atila Butkovits from comment #31)

Backed out for causing lint failure.

Backout link: https://hg.mozilla.org/integration/autoland/rev/fd4addd084c6dff9b50fe4da66d7262858b5d323

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=f5460317d2440c225f4a6942453749168765f089&selectedTaskRun=fqk3yORcRFWxAc44SIRfAQ.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=392240661&repo=autoland&lineNumber=112

== Change summary for alert #35640 (as of Fri, 07 Oct 2022 23:13:55 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
0.11%	installer size	osx-shippable	instrumented	121,817,623.25 -> 121,951,031.58

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=35640

Hi chunmin! Is it possible that the backout of this defect or 1793497 may have caused this slight increase in installer size ?

Comment 39

[C.M.Chang[:chunmin]]

Hi Alex, How did you tell the increased size is caused by that "backout"? Is there a way to verify it? I don't understand how "backout" can increase the installer size exactly. That "backout" removed all the code implemented in bug 1774300 and bug 1793497 in our codebase. If the "backout", which removes the codes from our codebase, increase the installer size, then it means putting those code in our codebase can have a smaller installer size. If "backout" was the cause of increased size, then it's not anymore. Those "backout" code has been landed in our code base again (comment 36), so it shouldn't be an issue now.

Comment 40

[Alex Finder|PTO, returning August 18th]

(In reply to C.M.Chang[:chunmin] from comment #39)

Hi Alex, How did you tell the increased size is caused by that "backout"? Is there a way to verify it? I don't understand how "backout" can increase the installer size exactly. That "backout" removed all the code implemented in bug 1774300 and bug 1793497 in our codebase. If the "backout", which removes the codes from our codebase, increase the installer size, then it means putting those code in our codebase can have a smaller installer size. If "backout" was the cause of increased size, then it's not anymore. Those "backout" code has been landed in our code base again (comment 36), so it shouldn't be an issue now.

After performing additional retriggers around that revision, it seems the spike was actually noise on the graph. Sorry for the confusion and the late reply, I'll close the regression and the alert as invalid.