1803353 - Add support for 10-bit YUV422 video or `yuv422p10le` format or `x422` codec

Status: NEW

(Core :: Audio/Video: Playback, enhancement, P3)

Description

[Paul Adenot (:padenot)]

Attachment: bad-chroma.webm

The file is from QuickTime Player on a Macbook M1 Max. I then exported it in ProRes in iMovie, this produced a file in yuv422p10le by default, using the following ffmpeg commands:

ffmpeg -i input.mov -b:v 800k -vf scale 1920:-1  -minrate 300k -maxrate 2610k -tile-columns 2 -g 240 -threads 8 -quality best -crf 22 -c:v libvpx-vp9 -c:a libopus   -pass 1 -speed 0 output.webm
ffmpeg -i input.mov -b:v 800k -vf scale 1920:-1  -minrate 300k -maxrate 2610k -tile-columns 3 -g 240 -threads 8 -quality best -crf 22 -c:v libvpx-vp9 -c:a libopus   -pass 2 -speed 0 -y output.webm

ffprobe says the following:

  Duration: 00:00:15.01, start: 0.258000, bitrate: 506 kb/s
  Stream #0:0: Video: vp9 (Profile 3), yuv422p10le(tv, bt709, progressive), 1920x1080, SAR 1:1 DAR 16:9, 29.97 fps, 29.97 tbr, 1k tbn (default)
    Metadata:
      HANDLER_NAME    : Core Media Video
      VENDOR_ID       : [0][0][0][0]
      TIMECODE        : 00:00:00:00
      ENCODER         : Lavc59.37.100 libvpx-vp9
      DURATION        : 00:00:15.005000000

I'm attaching two screenshots and an excerpt of the file. I only see this on macOS, current Nightly (same in current release, it doesn't play in previous releases).

Comment 1

[Paul Adenot (:padenot)]

Attachment: correct.png

Comment 2

[Paul Adenot (:padenot)]

Attachment: incorrect.png

Comment 3

[Paul Adenot (:padenot)]

It's confusing, but the video is itself a recording of Firefox -- the screenshots show the video rendering.

Comment 4

[Paul Adenot (:padenot)]

I noticed that for some reason this doesn't happen here on bugzilla, but slightly changing the DOM (putting the video element outside of the <span> it's in, or opening it in a different tab) reproduces the problem.

Comment 5

[Brad Werth [:bradwerth] | Out until Feb 2025]

Here's the file's MediaInfo output -- just the Video section. Since the bit depth isn't explicit here, I'm guessing that we extract it from the codec and then we make the wrong assumption about endian-ness. I'll figure out what's going on.

Video
ID                                       : 1
Format                                   : VP9
Codec ID                                 : V_VP9
Duration                                 : 15 s 5 ms
Bit rate                                 : 479 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 29.970 (30000/1001) FPS
Color space                              : YUV
Bits/(Pixel*Frame)                       : 0.008
Time code of first frame                 : 00:00:00:00
Time code source                         : Matroska tags
Stream size                              : 879 KiB (95%)
Title                                    : Core Media Video
Writing library                          : Lavc59.37.100 libvpx-vp9
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
VENDOR_ID                                : [0][0][0][0]

Comment 6

[Brad Werth [:bradwerth] | Out until Feb 2025]

Returning to this. The problem is that we unilaterally create 4:2:0 surfaces when a pref is set, and our 4:2:2 surface code is presuming 8-bit color depth. I'm not sure about the pref issue, but I think I can make the 4:2:2 surface creation work for 10-bit color.

Comment 7

[Brad Werth [:bradwerth] | Out until Feb 2025]

Attachment: WIP: Bug 1803353: Add support for macOS 422 10-bit video codec x422.

This isn't working yet. This attempts to clarify that the decision of
whether to create a 4:2:0 surface or a 4:2:2 surface should not be
controllable by a pref -- it's decided by the format of the video data.
This unfinished patch has incorrect planar info set on the IOSurface,
which causes severe screen-wide flashing visuals, similar to the comment
https://bugzilla.mozilla.org/show_bug.cgi?id=1657107#c22. When the
IOSurface properties are set correctly, we should be able to display this
content, possibly with bad tone-mapping since the 4:2:2 surfaces are not
eligble to become a TextureHost, which is what makes macOS put them in a
specialized video layer.

Comment 8

[Brad Werth [:bradwerth] | Out until Feb 2025]

I'm not at all certain I can fix this, so I'm removing myself from the Bug. Here's what I've learned.

The difference between YUV420 and YUV422 is minimal in that they are both planar formats, so our intuitively our handling of them should be closely related. But in macOS code at least we handle them very, very differently because we don't even define the planar information for YUV422 surfaces. It looks like instead we presume all YUV422 is the 8-bit APPLE_rgb_422 format and pack it into 4 byte elements and draw it specially in the shader. Presumably this was done to take advantage of some macOS or OpenGL fast paths. Making this more robust to handle higher bit depth YUV422 would entail either:

1. Removing this fake-RGB magic conversion and building a pipeline similar to how we handle NV12 (YUV420 8-bit) and P010 (YUV420 10-bit).
2. Leaving the YUV422 8-bit path alone and create differentiated enums in gfx::SurfaceFormat to represent the old 8-bit optimized path and a more generalized path that could handle 10-bit.

It's a heavy lift that shouldn't be treated as a "this particular video looks bad" Bug. I'm taking myself off this Bug and I'm going to recharacterize it as an enhancement, and put it through triage again.