1915265 - HDR detection broken on VP9 in mp4 container

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, defect)

Description

[Chris [:godtamit]]

Steps to reproduce:

1. Encode an HDR video with VP9 video and AAC audio in an MP4 container.

• Here's a video that can reproduce this: https://mega.nz/file/5lsGUYpT#B_KNF251rODrq5lWwGYHxSuBsiqEZQsGCRCWO83iAdY

2. Try to play back file in Firefox by pointing it at the file://

I suspect this is because of the assumption that all mp4 containers must be holding an H.264 encoded video:

• https://searchfox.org/mozilla-central/rev/6ad5adeba2b4353c53fd3c714223becd78cda029/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#1645

The assumption that an MP4 container always contains an H.264 encoded video is not always true.

Actual results:

Notice the video is washed out.

Expected results:

The video should play back normally. If repacking the video with VP9 + Opus in a webm container (direct video stream copy + audio transcode via ffmpeg -i <file_in> -c:v copy -c:a libopus <file_out>), notice HDR is working perfectly fine.

Comment 1

[Chris [:godtamit]]

Additional note: this was done on macOS. I don't think HDR video playback is working for any other platform.

Comment 2

[Brad Werth [:bradwerth]]

I can replicate. I'm not sure exactly where the fault is, but by the time we are displaying the video frame, we've erroneously concluded that it has BT709 color primaries and transfer function (rather than BT2020 / PQ).

Comment 3

[Brad Werth [:bradwerth]]

I think the problem is that VPXChangeMonitor::CheckForChange is willing to change the color space and bit depth, but not the color primaries or transfer function. A comment says that they can't change, but they appear to change with this video encoding. I think I can build a patch that handles this better.

Comment 4

[Brad Werth [:bradwerth]]

(In reply to Brad Werth [:bradwerth] from comment #3)

I think the problem is that VPXChangeMonitor::CheckForChange is willing to change the color space and bit depth, but not the color primaries or transfer function.

Hmm.... the info we have access to in VPXChangeMonitor::CheckForChange does not specific color primaries or the transfer function, so the original comment is likely correct. So why do we have bad initial data when we create the change monitor? Still looking...

Comment 5

[Brad Werth [:bradwerth]]

Hmm... not sure I can make any more progress on this. The MP4 decoder doesn't seem to even read color info in MP4Metadata::GetTrackInfo. It makes a call to mp4parse_get_track_video_info, but that doesn't return any color info. The bindgen file for that call defines a Mp4parseNclxColourInformation struct, but only AV1 decoder seems to fill it out.

I think this Bug is not for me. Kicking it back to the Media team for triage.

Comment 6

[Chris [:godtamit]]

This might be the relevant bug in mp4parse for this issue: https://github.com/mozilla/mp4parse-rust/issues/390

Comment 7

[Chris [:godtamit]]

Ah I've figured it out. You were on the right track for this:

Hmm.... the info we have access to in VPXChangeMonitor::CheckForChange does not specific color primaries or the transfer function, so the original comment is likely correct. So why do we have bad initial data when we create the change monitor? Still looking...

In VPXChangeMonitor's construction, those fields are never set (from parsing the VPCC), so they are default-initialized to None. I'll try and put a patch out today.

Comment 8

[Chris [:godtamit]]

Attachment: Bug 1915265 - Fix VP9 color info detection r=#media-playback-reviewers

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eb3547520039
Fix VP9 color info detection r=media-playback-reviewers,padenot

Comment 10

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/eb3547520039