1090232 - Track frame-time statistics during repeated tests on complex sites

Status: RESOLVED INCOMPLETE

(Datazilla Graveyard :: Metrics, defect)

Description

[Chris Lord [:cwiiis]]

Sorry if this is the wrong component or too vague, but here's the idea:

Currently, we test and track JavaScript performance, memory use, startup time and other such metrics, but we do so using benchmarks or in very isolated fashions. As reflected by personal experience and anecdotal 'evidence', this does not reflect the actual real-world performance of the browser (which I tend to find is a lot 'jankier' than Chrome and IE are right now).

We should try to measure some of the metrics that actually reflect how people feel the browser performs, and one of these is our frame-time during composition.

Benoit Girard's scrollgraph[1] is an excellent first step towards this, and I think we need to automate and track this.

I suggest that a combination of something like scrollgraph, as well as worst-case frame time and dropped frames, tracked over something that approximates 'typical' user input on a complex site, using something like marionette, would be a useful thing to have and eminently doable.

Filing this bug to get the discussion (and perhaps work?) going.

I'd normally say something like this would be invaluable for mobile, but as someone made the decision to include Super Hexagon in the Mozilla humble bundle, I'd say it's pretty important for desktop now too :p (the game requires consistent 60Hz feedback and shows up the jankiness in our browser terribly, even on my Core i7 Windows machine)

[1] http://benoitgirard.wordpress.com/2013/12/06/introducing-scroll-graph/

Comment 1

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Closing per https://bugzilla.mozilla.org/show_bug.cgi?id=1426399 as incomplete.