748143 - Should be able to run Eideticker tests without HDMI capture

Status: RESOLVED FIXED

(Testing Graveyard :: Eideticker, defect)

Description

[William Lachance (:wlach)]

Chris Lord wrote some great telemetry for measuring checkerboarding in bug 717349. For tackling bugs like bug 741988 where we only want to measure performance in fennec, it would be useful to be able to use this instead of the whole HDMI capture song and dance. It's much faster and doesn't require any specialized hardware.

In theory this duplicates some of the work already done on talos robocop tests. In practice, those are rather difficult to set up and run compared to the eideticker harness (which is exceedingly simple). I've done some initial prototyping and think I should be able to get something working smoothly by tomorrow. It should speed things along nicely.

Comment 1

[William Lachance (:wlach)]

(CCing various people who I think would be interested in this)

Comment 2

[William Lachance (:wlach)]

This is committed (). You can now run tests and measure checkerboarding using only internal telemetry like this (after setting up eideticker):

adb shell setprop log.tag.GeckoLayerRendererProf DEBUG
./bin/get-metric-for-build.py --no-capture --get-internal-checkerboard-stats nightly.apk  src/tests/scrolling/taskjs.org/index.html

Initial tests seem to indicate that the results are isomorphic to what you'd see with an HDMI capture, which is very good news.

Comment 3

[Mark Finkle (:mfinkle) (use needinfo?)]

I think adding the telemetry-based values as a sanity check to the video-based values is a good thing. It helps us have a better confidence in the Talos tests running in inbound and m-c. I don't think we should abandon the video-based tests though.

Comment 4

[William Lachance (:wlach)]

(In reply to Mark Finkle (:mfinkle) from comment #3)
> I think adding the telemetry-based values as a sanity check to the
> video-based values is a good thing. It helps us have a better confidence in
> the Talos tests running in inbound and m-c. I don't think we should abandon
> the video-based tests though.

Not at all. Both types of tests have their place. My main motivation writing this is to help devs debug specific problems with checkerboarding like bug 741988. You shouldn't need specialized hardware to do a bisection when we have the performance information available as internal telemetry.

Comment 5

[William Lachance (:wlach)]

Going to mark this as fixed, though we might want to tweak some of this in a follow up.