947314 - Need a reference for verifying performance measurement

Status: RESOLVED WONTFIX

(Testing Graveyard :: Eideticker, defect, P3)

Description

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Filing under Eideticker, though this might be more generally useful.

We should have a reference tool for validating Eideticker external measurement (or any other technique we use to do the same). That way we can better understand any potential inaccuracy, as well as verify the correctness of Eideticker's measurements after code changes.

The tool should be capable of reproducing the various scenarios Eideticker measures in a highly predictable fashion. The analogy would be having a tuning fork available to test the accuracy of an electronic tuner.

Currently the scenarios to reproduce would be:

fps -- display an animation (preferably a canned scrolling and/or panning animation, since that's what's most typically measured) at various speeds, like 5fps, 10fps, 15fps, 30fps, 60fps.

timetostableframe -- run through a number of variable frames for a set amount of time, then land on an unchanging frame. Amount of time can be varied.

I'll go out on a limb for a couple of suggestions for the tool name: Eidetuner or, my favorite, TuningFox.

Comment 1

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Adding the flashing corners demo as a dependency. Think we can probably figure out more tests too, so not closing out as a dupe yet.

Comment 2

[Mason Chang [Inactive] [:mchang]]

See also - 874781 - Should add flashing corners at every frame. If you can test that with a High Speed Camera, then we should be able to verify FPS.

Comment 3

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

(In reply to Mason Chang [:mchang] from comment #2)
> See also - 874781 - Should add flashing corners at every frame. If you can
> test that with a High Speed Camera, then we should be able to verify FPS.

Is that guaranteed to flash at 60 fps no matter what?

The flashing corners demo, for example, turns out not to run at an actual FPS at all. 

The rough conclusion I've come to so far is our reference will need to be built outside FxOS in native code with guaranteed framerate--since FxOS is the system under test, assuming it's stable enough to be a reference is problematic unless we identify an older version that's rock solid.

Comment 4

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

"at an actual 60 FPS" re: flashing corners. Problem there is that requestAnimationFrame doesn't run at 60fps reliably on Gecko on any platform, afaict.

Comment 5

[BMO Automation]

Eideticker has been discontinued, see bug 1361056