1995151 - Implement `rotation` and `flip` support in VideoFrame

Status: NEW

(Core :: Audio/Video: Web Codecs, task, P3)

Description

[C.M.Chang[:chunmin]]

Attachment: videoframe-rotation-visual.html

Overview

In the latest WebCodecs specification, the rotation and flip properties were added to the VideoFrame WebIDL to indicate the rotation and horizontal flip that should be applied when a VideoFrame is rendered. The flip is applied after the rotation.

VideoFrame

To comply with this update, two algorithms need to be introduced:

1. Parse Rotation
2. Add Rotations

These algorithms are executed by the VideoFrame constructors (See Chrome’s implementation here).

For VideoFrame(image, init) (code here), the path starts from Initialize Frame With Resource (code here, used when constructing from ImageElement, SVGImageElement, CanvasElement, OffscreenCanvasElement, or ImageBitmap), or Initialize Frame From Other Frame (code here, used when constructing from a VideoElement or another VideoFrame). Both paths eventually reach Initialize Visible Rect, Orientation, and Display Size (code here), where Parse Rotation and Add Rotations are executed. On the other hand, Parse Rotation can be called directly by VideoFrame(data, init) (code here).

ImageDecoder

These two properties should also be set when a VideoFrame is decoded via an ImageDecoder. In the Decode Complete Frame (code here) and Decode Progressive Frame algorithms, if the metadata includes orientation information, it should be reflected in the resulting VideoFrame.

Rendering

The most critical part of this implementation is rendering. Ideally, rotation and flip should be applied without requiring additional copies. This can be achieved if the renderer natively supports these transformations. Otherwise, rotated-then-flipped copies of the original image may be needed as rendering inputs. Libraries such as libyuv provide relevant APIs for this purpose:

• rotate_argb.h : ARGBRotate for RGBX images
• rotate.h : I420Rotate, RotatePlane, and RotateUV180 / RotateUV270 for YUV and NV formats
• planar_functions.h : I420Mirror and ARGBMirror for flipping (mirroring) images

The most common rendering API is likely be Canvas 2D's drawImage() (code here and spec here). We need to ensure that the drawn image reflects both rotation and mirroring correctly.

Testing & Validation

The expected outcome is to pass videoFrame-orientation.any.js. The image-decoder-image-orientation-none.https.html might also be relevant. Note that videoFrame-orientation.any.js validates only RGB formats.

To help with that, an additional visual test page (created using Google Gemini with necessary modifications) is attached. It allows testing rotation and flip behavior interactively, using sources from an image URL, a webcam capture, or a four-color square in RGBX, I420, or NV12 formats. This will significantly expand test coverage and help visually confirm correctness. Additionally, it would be beneficial if the I420 and NV12 format tests from the visual test page could be modified and integrated into the current WPT suite to expand test coverage across all browsers.