1311537 - Automated perf comparison between a11y and non-a11y modes of operation

Status: NEW

(Core :: Disability Access APIs, defect, P3)

Description

[(No longer employed by Mozilla) Aaron Klotz]

There obviously must be a performance penalty to using a11y. This is particularly noticeable during DOM mutation. Since a11y is increasingly being enabled due to touch screens and the like, we need to know more about how it affects everyday browser perf.

We don't really have good data right now on this subject, but we'd like to develop a test to quantify user-noticeable performance changes.

One idea that was suggested was simply to run some talos suites with a11y on and with a11y off and comparing them, but we're not sure whether that would be meaningful.

Comment 1

[(No longer employed by Mozilla) Aaron Klotz]

Joel, Avi, do you have any suggestions about how we should go about testing a11y vs non-a11y performance?

Comment 2

[alexander :surkov (:asurkov)]

On a technical side it's enough to request nsIAccessibilityService service to start the accessibility engine, it will stay alive until Firefox is closed

Comment 3

[Avi Halachmi (:avih)]

(In reply to Aaron Klotz [:aklotz] from comment #1)
> Joel, Avi, do you have any suggestions about how we should go about testing
> a11y vs non-a11y performance?

We should probably start with try pushes with of all tests enabled - with and without a11y enabled, and compare the results via perfherder, however I don't know how to force enable/disable a11y.

Also, I haven't been involved and therefore unfamiliar with the specifics of the actual talos a11y test.

(In reply to alexander :surkov from comment #2)
> On a technical side it's enough to request nsIAccessibilityService service
> to start the accessibility engine, it will stay alive until Firefox is closed

I think we'd need a deterministic way to both enable and disable a11y (it's fine if it's for the entire Firefox session), such that the comparison would work as expected on:
- Platforms/systems which enable a11y by default
- Platforms/systems which disable a11y by default

It would be easiest to control if there was a pref which could force enable/disable it.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I think the best way to look at this is the e10s dashboard:
https://treeherder.mozilla.org/perf.html#/e10s?filter=a11y&timerange=604800

it shows that we are pretty close (except for linux64) using the a11y talos test that we run.

Comment 5

[Trevor Saunders (:tbsaunde)]

(In reply to Joel Maher ( :jmaher) from comment #4)
> I think the best way to look at this is the e10s dashboard:
> https://treeherder.mozilla.org/perf.html#/e10s?filter=a11y&timerange=604800

I'm not completely sure what question we want to answer here, but I think it more like how does a11y compare to not a11y than how does e10s a11y compare to not e10s a11y.

> it shows that we are pretty close (except for linux64) using the a11y talos
> test that we run.

well, sort of the full picture is a bit more interesting than that.  the dhtml test was an across the board improvement by around 10%.  Which off hand doesn't make any sense to me, but it seems to be true.  On the other hand there's a pretty significant regression accross the board for the table test, but for some reason its much worse on linux 64.

Comment 6

[Trevor Saunders (:tbsaunde)]

(In reply to Aaron Klotz [:aklotz] from comment #0)
> There obviously must be a performance penalty to using a11y. This is
> particularly noticeable during DOM mutation. Since a11y is increasingly
> being enabled due to touch screens and the like, we need to know more about
> how it affects everyday browser perf.

What clients are using a11y probably has a significant effect on the exact work load.

> We don't really have good data right now on this subject, but we'd like to
> develop a test to quantify user-noticeable performance changes.

do you mean for cases where accessibility is enabled and nothing happens with it, or some but not much happens or its heavily used? or some combination of those?

> One idea that was suggested was simply to run some talos suites with a11y on
> and with a11y off and comparing them, but we're not sure whether that would
> be meaningful.

I expect it is an accurate comparison of a11y is enabled but not used for much of anything to a11y is not enabled.  I suspect some tests may not be effected if they spend most of there time painting or something, but if there are tests that are different, and we assume we are only running useful tests then I expect there is value in tracking the difference between a11y and not a11y for those tests.

Comment 7

[Julien Cristau [:jcristau]]

Mass wontfix for bugs affecting firefox 52.