2021178 - make buffer size validation functions in gfx/layers/ImageDataSerializer.h return Maybe<>

Status: RESOLVED FIXED

(Core :: Graphics, enhancement)

Description

[Timothy Nikkel (:tnikkel)]

This is a followup to a review comment in bug 2020091.

We use various functions in ImageDataSerializer.h to validate buffer sizes (ComputeYCbCrBufferSize, ComputeRGBBufferSize, GetRGBStride, ComputeRGBStride, etc) which return 0 on error/overflow. Historically the code was not checking for 0 return value in some places (I already did an audit and fixed anything that didn't seem right). We should make it harder to not check for the error condition to avoid this happening again. Further, it was suggested we add logging/assertions in the error path.

Comment 1

[Tom Schuster [:tschuster]]

Attachment: (secure)

Comment 2

[Tom Schuster [:tschuster]]

I just updated all of the GetAligned functions to return Maybe<> and there seem to be quite a few places where we weren't checking for 0 at all.

Comment 3

[Tom Schuster [:tschuster]]

Attachment: (secure)

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/1dc03331dd13
https://hg.mozilla.org/integration/autoland/rev/96617faea688
Return Maybe<uint32_t> from Compute*BufferSize. r=tnikkel,media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/601fafa65ba6
https://hg.mozilla.org/integration/autoland/rev/1ec4feda5059
Return Maybe<int32_t> from Get*Stride. r=tnikkel,media-playback-reviewers,webrtc-reviewers,webgpu-reviewers,bradwerth,ErichDonGubler,pehrsons,padenot

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/96617faea688
https://hg.mozilla.org/mozilla-central/rev/1ec4feda5059