1425311 - Add pageloader support for multiple hero measurements per page

Status: RESOLVED WORKSFORME

(Testing :: Talos, enhancement)

Description

[Robert Wood [:rwood]]

Currently talos pageloader tests measure one (and only one) specific event, i.e. moz after paint, time to first non-blank paint, hero element (landing soon in Bug 1418368).

Add the ability to measure more than one item in a single pageload. For example, have tp6_google measure both fnbpaint and hero element in each pageload.

Treat these values as typical talos subtests - i.e. the top level test result/score (used for alerting) will be the median of these two items. However the individual measurement values will be available in the subtests view in perfhereder compare, as well as in the perfherder-data.json (and local.json when talos is run locally) files.

Comment 1

[Robert Wood [:rwood]]

Note: It is up for discussion re: subtests, we could also have all the measured values reported in such a way that they appear to be separate tests, even though they were produced by the single test run.

Comment 2

[Robert Wood [:rwood]]

Attachment: Bug 1425311 - Add new talos test measurements settings, to define what the test is measuring. Use this setting instead of individual keys. This will allow us to specifiy multiple measurement keys in a single test, in the future.

Review commit: https://reviewboard.mozilla.org/r/207878/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/207878/

Comment 3

[Robert Wood [:rwood]]

Hey Tarek, regarding the hero element you added to tp6_google (elementtiming="hero").

Since the observer in lh_hero.js is watching the element found via querySelector of the attribute name ['elementtiming'], I believe if there are multiple hero elements in a single page, it's the element attribute name 'elementtiming' itself that would be different for each hero element, correct?

OR do hero elements always have the attribute name ('elementiming') but the value (i.e. 'hero') can be different for each hero element? In which case we'd have to use querySelector to search for an element with that attribute AND corresponding value, when defining which element to observe. Thanks!

Comment 4

[Robert Wood [:rwood]]

I think I've answered my own question now, looks like it will always be 'elementtiming' but the value will change for each hero element - and querySelector with an '[attr=val]' selector can be used for each unique hero element that tp6 is measuring.

Comment 5

[Robert Wood [:rwood]]

Attachment: Bug 1425311 - Add pageloader support for multiple hero measurements per page (WIP)

Review commit: https://reviewboard.mozilla.org/r/218572/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/218572/

Comment 6

[Robert Wood [:rwood]]

This is supported in Raptor (under development). I don't see the need to also add this to Talos pageloader, so I'd like to close this bug as resolved won't fix, if you agree :jmaher.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

agreed