Add a nav-bench variant with accessibility enabled
Categories
(Testing :: Raptor, task, P3)
Tracking
(Not tracked)
People
(Reporter: acreskey, Assigned: acreskey)
References
(Blocks 2 open bugs)
Details
(Whiteboard: [fxpe])
Attachments
(1 external file)
Add a variant of the navigation benchmark (browsertime-nav-bench.toml) that runs with accessibility enabled, so we can measure navigation performance when accessibility mode is enabled.
nav-bench is currently scored on visual metrics (SpeedIndex). With accessibility enabled, visual completeness does not capture the cost of building the accessibility tree, so the variant will probably also need to wait for the DOCUMENT_LOAD_COMPLETE accessibility event before stopping each measurement, and possibly report the time to that event as an additional metric.
Open question:
- Whether the variant should be scored on
SpeedIndex, on time toDOCUMENT_LOAD_COMPLETE, or both.
| Assignee | ||
Updated•5 days ago
|
Updated•5 days ago
|
| Assignee | ||
Updated•5 days ago
|
| Assignee | ||
Comment 1•5 days ago
|
||
nav-bench-a11y runs the nav-bench pages with accessibility.force_disabled=-1,
which starts the accessibility service at browser startup. Before the first
navigation the test waits for the service to be running, starting it
explicitly if the pref did not.
After each measured navigation, the test also waits for the top-level
document's EVENT_DOCUMENT_LOAD_COMPLETE and reports the time from the
document's performance.timeOrigin to that event as a
<site>-<phase>-a11y-doc-load-complete subtest.
Comment 2•4 days ago
|
||
(In reply to Andrew Creskey [:acreskey] from comment #0)
nav-bench is currently scored on visual metrics (
SpeedIndex). With accessibility enabled, visual completeness does not capture the cost of building the accessibility tree, so the variant will probably also need to wait for theDOCUMENT_LOAD_COMPLETEaccessibility event before stopping each measurement, and possibly report the time to that event as an additional metric.Open question:
- Whether the variant should be scored on
SpeedIndex, on time toDOCUMENT_LOAD_COMPLETE, or both.
Thanks for filing this :)
Ideally we'd use something like "time to first speech" to approximate when information is ready for consumption by screen reader users, in the same way that the visual metrics approximate when info is ready for consumption by sighted users.
Running ATs on CI is complicated, though :) so we could instead consider "the information is ready for ATs to query, in the parent process" as our completion metric. This roughly equates to "the initial cache push has been received". The cache receiving happens here, and we've already got a profile marker/perfstat for it.
The initial cache push will contain all info a11y can compute about the page from its starting state, but if the page keeps changing the data in the cache will become outdated. We'll continue to update the cache with subsequent cache pushes, or add new information if new DOM nodes/frames are added.
If the page settles visually we should see these pushes settle too.
I have some try runs going to collect data on how EVENT_DOCUMENT_LOAD_COMPLETE, initial cache push, update pushes, and mutation events line up with the visual metrics SpeedIndex is currently using. I'll share them here once I've got them in a readable form
Comment 3•4 days ago
|
||
Looking into SpeedIndex some more, I see this line in the docs:
Calculated by using videos of the pageload which provide a measure of visual completeness. Visual completeness is calculated by comparing the histogram of a given frame to the final frame of the pageload.
What determines "pageload" here?
| Assignee | ||
Comment 4•3 days ago
|
||
(In reply to Morgan Reschenberg [:morgan] from comment #3)
Looking into
SpeedIndexsome more, I see this line in the docs:Calculated by using videos of the pageload which provide a measure of visual completeness. Visual completeness is calculated by comparing the histogram of a given frame to the final frame of the pageload.
What determines "pageload" here?
For nav-bench, we are looking at the navigation as the user experiences it visually.
i.e. we record video and calculate the SpeedIndex,, how long it took to display the primary content.
In the wip patches I've tried to wait for the EVENT_DOCUMENT_LOAD_COMPLETE event however it doesn't fire for all navigations, so I will have to figure out why.
By the way, green CI run with profiles from the wip a11y navbench
here
Description
•