1578045 - Matrix4x4::TransformAndClipBounds does not clip bounds

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Markus Stange [:mstange]]

    Rect r = Matrix4x4().TransformAndClipBounds(Rect(0, 0, 2000, 50),
                                                Rect(1900, 100, 100, 100));

I would expect the rectangle r to be empty, because the to input rectangles do not intersect. But the return value is non-empty, and roughly equal to the first argument:

Rect(0.000000, 0.000000, 1999.999878, 50.000000)

Comment 1

[Markus Stange [:mstange]]

This was caused by attachment 8764294 [details] [diff] [review]. When the loop is exited early if the polygon has no more points, then the calculation size_t srcPointCount = dstPoint - points[0]; outside of the loop can be wrong: &points[0][0] is only the start of the current set of polygon points if the index of the most recently-applied clipping plane is odd (1 or 3). If that index is even (0 or 2), the current set of polygon points starts at &points[1][0].
Or in other words, this function does some double buffering and then assumes one of the buffers is in use when the loop is exited, but due to early exits it can be the other buffer that's currently in use.

Comment 2

[Markus Stange [:mstange]]

A more accurate bug summary is: If the clip rect's left or top edge causes the polygon to be clipped away completely, a non-empty polygon is returned which ignores that left or top clip edge, and also includes the point (0, 0). (The left edge's clipping plane is stored in planeNormals[0], the top edge's clipping plane is stored in planeNormals[2], and the bug only appears if the loop is exited at plane index 0 or 2. And the wrongly constructed range of points happens to include all 32 points in points[0], which usually include ones that have been default-initialized to (0, 0) by the Point4D constructor.)

Comment 3

[Markus Stange [:mstange]]

Attachment: Bug 1578045 - Correctly return zero vertices if clipping plane 0 or 2 clip away the entire polygon. r=kip,mattwoodrow

Comment 4

[Markus Stange [:mstange]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1&revision=d66393bd59b15702b74cc5826863d9695a4205a2

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9e38d12a61b2
Correctly return zero vertices if clipping plane 0 or 2 clip away the entire polygon. r=kip

Comment 6

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/9e38d12a61b2