1381037 - Remove the CART Talos test when Nightly becomes 57

Status: RESOLVED FIXED

(Testing :: Talos, enhancement)

Description

[Mike Conley (:mconley) (:⚙️)]

The CART test exercises the customize mode transition that was introduced in 2014 with the Australis theme refresh.

The Photon re-theme effort removed that animation in bug 1354123[1], so when 57 rolls around with the Photon theme, this test will no longer be useful.

The CART test is still returning values, but I don't suspect they have any meaning. We should probably treat any regressions on CART with extreme skepticism if Photon is enabled.

[1]: Unsurprisingly, we saw a dramatic improvement on the CART test when we removed the animation: Bug 1354123 comment 11.

Comment 1

[Mike Conley (:mconley) (:⚙️)]

(In reply to Mike Conley (:mconley) - Digging out of needinfo / review backlog. from comment #0)
> The CART test is still returning values, but I don't suspect they have any
> meaning. We should probably treat any regressions on CART with extreme
> skepticism if Photon is enabled.
> 

Hey igoldan, just needinfo'ing you to give you a heads-up on the above.

Comment 2

[Mike Conley (:mconley) (:⚙️)]

jaws mention in fx-team that the idea of CART has some utility. Reparenting all of the toolbar items as customizable items does take work, and it might be useful to capture that.

I don't think the TART framework (which CART uses) is the right choice for that though, tbh. The TART framework uses the frame timing API to measure how long it takes to paint each frame, and also tries to capture a delta between how long the animation is _supposed_ to be (right now that's 0), and how long the animation is (right now, that's also 0).

So perhaps we should replace the test with something better. I still will say, however, that CART results as they stand are not useful in the brave new Photon world, in their current form.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I agree we should not run the test; ideally replace it with something new that is meaningful.