1765204 - Compositor hit test can target incorrect process in the presence of perspective transforms

Status: VERIFIED FIXED

(Core :: Panning and Zooming, defect, P2)

Description

[Botond Ballo [:botond]]

Spinoff from bug 1745834, to track the second issue that was not fixed in that bug.

STR

1. Run a recent Firefox nightly that contains the fix for bug 1745834
2. Load https://support.2vr.at/v60/external/
3. Click and drag the panorama close to, but not in the pinned iframe.


Actual results

Dragging has no effect when starting close to the iframe. If you start the drag further away from the iframe, it works.

Expected results

Dragging should work even when starting close to (but not inside) the iframe.

Comment 1

[Botond Ballo [:botond]]

Attachment: Reduced testcase

Copying over the reduced testcase and STR for it:

STR

Move the mouse around, observing the behaviour when the mouse is close to the iframe and when it's farther from it.

Expected results

The red circle follows the mouse whenever it's outside the iframe, including when it's close to the iframe.

Actual results

There is a region larger than the iframe, where the red circle does not follow the mouse.

Basically, in this boundary region, events that should target the containing document are incorrectly targeting the iframe instead.

Comment 2

[Botond Ballo [:botond]]

I haven't gotten to the bottom of this, but the high-level picture I'm starting to build is that:

• we ship un-flattened transforms to WebRender
• WebRender has its own logic for flattening them (see e.g. TransformStyle::Flat and CoordinateSystem::should_flatten)
• different codepaths are taken to apply transforms during rendering (mainly get_relative_transform()) and hit-testing (get_world_transform()), yielding different results such that this page renders correctly but hit-tests incorrectly

cc Glenn as it seems like this may be related to bug 1532174 which refactored some of these codepaths.

Comment 3

[Botond Ballo [:botond]]

Ok, I believe I've figured this out.

The difference in behaviour between rendering and hit-testing isn't related to "flattening" (in the sense of handling TransformStyle::Flat) per se, but rather a consequence of the fact that the two operations apply the transform in different directions: rendering applies the transform to a primitive to get its coordinates in screen space, and hit-testing applies the inverse of the transform to coordinates in screen space to see if a primitive was hit.

Round-tripping a point on a primitive through these two transformation should give us back the original point, but with 3D transforms extra care is required to make sure this happens.

In this testcase, we have a 3D transform represented as the following 4x4 matrix:

[  1.0,   0.0,    0.0,    0.0;
   0.0,   1.0,    0.0,    0.0;
  -0.6,  -0.6,    2.0, -0.002;
 200.0, 200.0, -500.0,    1.0]

Let's say we're applying this to a starting point (200, 200) on our primitive. To do this, we extend the 2D point (200, 200) with z=0 and w=1 to get the 4D point (200, 200, 0, 1). Then we perform matrix multiplication, to get the 4D point (400, 400, -500, 1). Finally, we project this point to 2D by dropping z and dividing x and y by w, to get (400, 400).

Note that to go from 2D to 4D, we extended with z=0, but to go from 4D back to 2D, we dropped a z=-500.

Now let's try hit-testing screen coordinates at (400, 400) to see if we get back our original point of (200, 200).

If we use the same procedure as before, that is, extend the (400, 400) to (400, 400, 0, 1) and then multiply the inverse of the above matrix, the resulting 4D point will be (300, 300, 500, 2), which projected to 2D is (150, 150) - different from our input point!

This makes sense if you think of it at the level of 4D vectors: if we want to get our input 4D vector, (200, 200, 0, 1), back, we better use the original result, (400, 400, -500, 1), as the input to the multiplication by the inverse -- not (400, 400, 0, 1).

The C++ hit-testing code in APZ gets this right: when applying a transform in the screen-to-primitive direction, it uses Matrix4x4Typed::ProjectPoint(), which works as follows:

  template <class F>
  Point4DTyped<TargetUnits, F> ProjectPoint(
      const PointTyped<SourceUnits, F>& aPoint) const {
    // Find a value for z that will transform to 0.

    // The transformed value of z is computed as:
    // z' = aPoint.x * _13 + aPoint.y * _23 + z * _33 + _43;

    // Solving for z when z' = 0 gives us:
    F z = -(aPoint.x * _13 + aPoint.y * _23 + _43) / _33;

    // Compute the transformed point
    return this->TransformPoint(
        Point4DTyped<SourceUnits, F>(aPoint.x, aPoint.y, z, 1));
  }

Notice how it computes a z value for the input 4D point that ensures we get a result 4D point with z=0 (e.g. in our case it will calculate and use z=-500 as input to the 4D vector to use for the multiplication).

However, the WebRender hit-test code (which is around here) does no such thing, and therefore arrives at the incorrect result.

Comment 4

[Botond Ballo [:botond]]

Attachment: Bug 1765204 - Add a mochitest. r=tnikkel

Comment 5

[Botond Ballo [:botond]]

Attachment: Bug 1765204 - Add a project_point2d operation and use it for hit testing. r=gw

Depends on D146368

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4a19d217b980
Add a mochitest. r=tnikkel
https://hg.mozilla.org/integration/autoland/rev/791f3d9fb42f
Add a project_point2d operation and use it for hit testing. r=gw

Comment 7

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/4a19d217b980
https://hg.mozilla.org/mozilla-central/rev/791f3d9fb42f

Comment 9

[Giorgia Nichita, Release Desktop QA]

Reproduced this issue on an affected Nightly build from 2022-04-18, on macOS 10.15.
Verified as fixed on Firefox 102.0b4 (20220605185654) on macOS 10.15, Win 10 x64 and Ubuntu 21.04.