Closed Bug 1578045 Opened 3 months ago Closed 3 months ago

Matrix4x4::TransformAndClipBounds does not clip bounds

Categories

(Core :: Graphics, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
mozilla71
Tracking Status
firefox-esr60 --- wontfix
firefox-esr68 --- wontfix
firefox69 --- wontfix
firefox70 --- wontfix
firefox71 --- fixed

People

(Reporter: mstange, Assigned: mstange)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file)

    Rect r = Matrix4x4().TransformAndClipBounds(Rect(0, 0, 2000, 50),
                                                Rect(1900, 100, 100, 100));

I would expect the rectangle r to be empty, because the to input rectangles do not intersect. But the return value is non-empty, and roughly equal to the first argument:

Rect(0.000000, 0.000000, 1999.999878, 50.000000)

This was caused by attachment 8764294 [details] [diff] [review]. When the loop is exited early if the polygon has no more points, then the calculation size_t srcPointCount = dstPoint - points[0]; outside of the loop can be wrong: &points[0][0] is only the start of the current set of polygon points if the index of the most recently-applied clipping plane is odd (1 or 3). If that index is even (0 or 2), the current set of polygon points starts at &points[1][0].
Or in other words, this function does some double buffering and then assumes one of the buffers is in use when the loop is exited, but due to early exits it can be the other buffer that's currently in use.

Regressed by: CVE-2016-5252

A more accurate bug summary is: If the clip rect's left or top edge causes the polygon to be clipped away completely, a non-empty polygon is returned which ignores that left or top clip edge, and also includes the point (0, 0). (The left edge's clipping plane is stored in planeNormals[0], the top edge's clipping plane is stored in planeNormals[2], and the bug only appears if the loop is exited at plane index 0 or 2. And the wrongly constructed range of points happens to include all 32 points in points[0], which usually include ones that have been default-initialized to (0, 0) by the Point4D constructor.)

Attachment #9089687 - Attachment description: Bug 1578045 - Correctly return zero vertices if clipping plane 0 or 2 clip away the entire polygon. → Bug 1578045 - Correctly return zero vertices if clipping plane 0 or 2 clip away the entire polygon. r=kip,mattwoodrow
Pushed by mstange@themasta.com:
https://hg.mozilla.org/integration/autoland/rev/9e38d12a61b2
Correctly return zero vertices if clipping plane 0 or 2 clip away the entire polygon. r=kip
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
You need to log in before you can comment on or make changes to this bug.