2057759 - Frames decoded in the RDD process lose their color space metadata when sent over the video bridge

Status: NEW

(Core :: Audio/Video: Playback, defect, P3)

Description

[David Torcivia]

Every video frame decoded in the RDD process and sent to the content process over the video bridge loses its color space metadata: RemoteVideoDecoderParent::ProcessDecodedData only fills YUVColorSpace, primaries, transfer function, and range in the shmem fallback path, and the texture path (which is what runs in practice on all desktop platforms) sends the defaults. The content side's GuessColorSpace for GPUVideoImage then reconstructs a color space from those defaults, which currently aliases to bt709 regardless of the stream.

Visible effect: VideoFrame.colorSpace in WebCodecs reports guessed values instead of the decoder's, for every software-decoded frame on Linux, Windows, and macOS (verified against current nightlies on all three). Chrome reports the decoder's values. This is the metadata sibling of bug 1969762 (which covers the pixel format and copyTo; see comment 9 there for the mechanics).

The one-line fix (hoist the extraction above the texture/shmem fork) is straightforward, but I prototyped it while working on bug 1969762 and it cannot land alone: forwarding the decoder's real values turns webcodecs/full-cycle-test.https.any.js?av1 "w/ stripped color space" red, because

1. our encoders write no color info into the bitstream (bug 2057758), so
2. the decoder falls back to display-size heuristics (DAV1DDecoder and VPXDecoder label SD streams BT601/BT470BG; the heuristic-primaries problem is bug 1869825), and
3. the test then correctly observes that the decoded metadata contradicts the encoder's claimed color space.

In other words the test passes today only because the metadata is lost in transit. Landing order that keeps everything green and honest: bug 2057758 first, then this forwarding, with the heuristic-vs-signaled distinction from bug 1869825 (or a signaled-only forwarding rule) resolving the remaining canvas-sourced cases.

Comment 1

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Test remote decoded color metadata transport

Comment 2

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Preserve decoded color metadata across remote image transport

Comment 3

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Preserve FFmpeg decoded frame color metadata

Comment 4

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Test FFmpeg decoded frame color metadata

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9e390b615c99
https://hg.mozilla.org/integration/autoland/rev/1206edfa90a1
Test remote decoded color metadata transport r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/9f52281a493b
https://hg.mozilla.org/integration/autoland/rev/039cd2f576eb
Preserve decoded color metadata across remote image transport r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/f34647f7ad7b
https://hg.mozilla.org/integration/autoland/rev/d6ffec2b6550
Test FFmpeg decoded frame color metadata r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/05cdb23a85d5
https://hg.mozilla.org/integration/autoland/rev/977a7ba20978
Preserve FFmpeg decoded frame color metadata r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/4401096dd365
https://hg.mozilla.org/integration/autoland/rev/7a279df68901
apply code formatting via Lando

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/5b0924fd3114
https://hg.mozilla.org/integration/autoland/rev/83742e9e5d33
Revert "Bug 2057759: apply code formatting via Lando" for causing build bustages @ FFmpegVideoDecoder

Comment 7

[amarc]

Backed out for causing build bustages

• Backout link
• Push with failures
• Failure Log

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/ff0070161e7d
https://hg.mozilla.org/integration/autoland/rev/284992548bf7
Test remote decoded color metadata transport r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/d421c372bc19
https://hg.mozilla.org/integration/autoland/rev/facb2d2ab204
Preserve decoded color metadata across remote image transport r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/c33eae190673
https://hg.mozilla.org/integration/autoland/rev/faff2953a251
Test FFmpeg decoded frame color metadata r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/8898eed6d09b
https://hg.mozilla.org/integration/autoland/rev/25562420acee
Preserve FFmpeg decoded frame color metadata r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/3090db88d357
https://hg.mozilla.org/integration/autoland/rev/b711862e4821
apply code formatting via Lando

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/c9a0c82e582e
https://hg.mozilla.org/integration/autoland/rev/2d9eabacbd54
Revert "Bug 2057759: apply code formatting via Lando" for causing reftest failure 720p.png.bt709.bt709.pc.yuv420p.av1.webm / 720p.png

Comment 10

[Norisz Fay [:noriszfay]]

Backed out for causing reftest failure 720p.png.bt709.bt709.pc.yuv420p.av1.webm / 720p.png

Backout link

Push with failures

Failure log

Comment 11

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Update draw-snapshot AV1 color-quads fuzziness

Preserving FFmpeg's decoded color metadata reduces the pixel differences for the full-range 8-bit and 10-bit AV1 cases. Update their draw-snapshot expectations to match the corrected output.

Comment 12

[json.hansen]

macOS, Firefox 155. Colors are washed out in video. Setting media.rdd-process.enabled=false immediately restores correct colors. Firefox 154 did not have the problem. Safari renders the same video correctly.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/01c45e1d0b91
https://hg.mozilla.org/integration/autoland/rev/a55bf6b19120
Test remote decoded color metadata transport r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/37d56c6540e4
https://hg.mozilla.org/integration/autoland/rev/4e1023a31f03
Preserve decoded color metadata across remote image transport r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/4f2697323872
https://hg.mozilla.org/integration/autoland/rev/0525781bc921
Test FFmpeg decoded frame color metadata r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/8765bd0c42d3
https://hg.mozilla.org/integration/autoland/rev/6fc26ec48e18
Preserve FFmpeg decoded frame color metadata r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/bb863e4e7ac2
https://hg.mozilla.org/integration/autoland/rev/26ece526d91a
Update draw-snapshot AV1 color-quads fuzziness r=media-playback-reviewers,alwu

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/edb890e8a3e4
https://hg.mozilla.org/integration/autoland/rev/7dd0886c03a3
Revert "Bug 2057759 - Update draw-snapshot AV1 color-quads fuzziness r=media-playback-reviewers,alwu" for causing failures at full-cycle-test.https.any.worker.html?av1.

Comment 15

[Atila Butkovits]

Backed out for causing failures at full-cycle-test.https.any.worker.html?av1.

Backout link: https://hg.mozilla.org/integration/autoland/rev/7dd0886c03a3

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=26ece526d91afc480b2106d66624d7dfa553e08c&selectedTaskRun=VvPut4PySuOy3HW42m8PKw.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=589620433&repo=autoland&task=VvPut4PySuOy3HW42m8PKw.0&lineNumber=1782

Comment 16

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Add debugging logs for PlanarYCbCrData

Comment 17

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Logging by gfx's logging convention

Comment 18

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Log colorspace in full-cycle path

Comment 19

[C.M.Chang[:chunmin]]

Attachment: Bug 2057759 - Gate decoded frame color metadata behind a preference r?#media-playback-reviewers

Keep the existing behavior by default until encoder color-space signaling is complete. Enable the new behavior for the color-quads reftests.