Open Bug 778493 Opened 8 years ago Updated 4 years ago

Add accessibility coverage for Windows tp tests on mc

Categories

(Testing :: Mochitest, defect, P2)

x86_64
Windows 7
defect

Tracking

(Not tracked)

People

(Reporter: jimm, Unassigned)

References

Details

(Whiteboard: feature=external)

On Windows touch enabled devices accessibility is loaded by default. We currently don't run talos tp tests though with accessibility. Currently the number of machines out there with touch input is low but that will likely change with the release of win8.

I did quick test on try server with a patch that enabled a11y by forcing the load of the root accessible object, the difference in tp is noticeable -

accessibility:

tp5r_paint: 368.73
tp5r_modlistbytes_paint: 60.6MB
tp5r_%cpu_paint: 76.42
tp5r_content_rss_paint: 76.4B
tp5r_main_rss_paint: 76.4B
tp5r_pbytes_paint: 165.5MB
tp5r_responsiveness_paint: 518.0
tp5r_shutdown_paint: 1356.0

without:

tp5n_paint: 275.03
tp5n_modlistbytes_paint: 59.5MB
tp5n_%cpu_paint: 71.81
tp5n_content_rss_paint: 71.7B
tp5n_main_rss_paint: 71.7B
tp5n_pbytes_paint: 128.4MB
tp5n_responsiveness_paint: 871.0
tp5n_shutdown_paint: 1318.0

Without test coverage here we can't work toward lower drag from the a11y code, so I'd like to propose we either run all our tp tests with a11y forced on, or run a new set of tp tests with it on.

Note I didn't see any appreciable difference in other talos tests in my try run. The results are here - 

https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884

We can control the forcing of a11y on through a pref.
(In reply to Jim Mathies [:jimm] from comment #0)
> On Windows touch enabled devices accessibility is loaded by default. We
> currently don't run talos tp tests though with accessibility. Currently the
> number of machines out there with touch input is low but that will likely
> change with the release of win8.
> 
> I did quick test on try server with a patch that enabled a11y by forcing the
> load of the root accessible object, the difference in tp is noticeable -
> 
> accessibility:
> 
> tp5r_paint: 368.73
> tp5r_modlistbytes_paint: 60.6MB
> tp5r_%cpu_paint: 76.42
> tp5r_content_rss_paint: 76.4B
> tp5r_main_rss_paint: 76.4B
> tp5r_pbytes_paint: 165.5MB
> tp5r_responsiveness_paint: 518.0
> tp5r_shutdown_paint: 1356.0
> 
> without:
> 
> tp5n_paint: 275.03
> tp5n_modlistbytes_paint: 59.5MB
> tp5n_%cpu_paint: 71.81
> tp5n_content_rss_paint: 71.7B
> tp5n_main_rss_paint: 71.7B
> tp5n_pbytes_paint: 128.4MB
> tp5n_responsiveness_paint: 871.0
> tp5n_shutdown_paint: 1318.0

that's noticable, but I'm actually sort of happy to see it isn't worse.  I'm also a little suprised those are the only tests effected.

> Without test coverage here we can't work toward lower drag from the a11y
> code, so I'd like to propose we either run all our tp tests with a11y forced
> on, or run a new set of tp tests with it on.

this seems like a good idea, but I'm not sure exactly which approach is best.

> We can control the forcing of a11y on through a pref.

yes, or from chrome code we can get nsIAccessibleRetrieval
> Note I didn't see any appreciable difference in other talos tests in my try
> run. The results are here - 
> 
> https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884

not really related, but we might want to atleast check correctness tests aren't effected too.
(In reply to Trevor Saunders (:tbsaunde) from comment #2)
> > Note I didn't see any appreciable difference in other talos tests in my try
> > run. The results are here - 
> > 
> > https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884
> 
> not really related, but we might want to atleast check correctness tests
> aren't effected too.

By correctness tests are you referring to reftests?
(In reply to Jim Mathies [:jimm] from comment #3)
> (In reply to Trevor Saunders (:tbsaunde) from comment #2)
> > > Note I didn't see any appreciable difference in other talos tests in my try
> > > run. The results are here - 
> > > 
> > > https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884
> > 
> > not really related, but we might want to atleast check correctness tests
> > aren't effected too.
> 
> By correctness tests are you referring to reftests?
I'd think only mochitests and reftests could reasonably be effected.
(In reply to Trevor Saunders (:tbsaunde) from comment #5)
> (In reply to Jim Mathies [:jimm] from comment #3)
> > (In reply to Trevor Saunders (:tbsaunde) from comment #2)
> > > > Note I didn't see any appreciable difference in other talos tests in my try
> > > > run. The results are here - 
> > > > 
> > > > https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884
> > > 
> > > not really related, but we might want to atleast check correctness tests
> > > aren't effected too.
> > 
> > By correctness tests are you referring to reftests?
> I'd think only mochitests and reftests could reasonably be effected.

so, yes, reftests and mochitests was what I meant
would you want to replace tp5n with tp5a (a11y version)?  Is there value in running this without a11y turned on?  

For reference tp5r above is much different than tp5n.  tp5n has removed all outside network calls and 404s.  It would be better to compare tp5n with and without a11y.
(In reply to Joel Maher (:jmaher) from comment #7)
> would you want to replace tp5n with tp5a (a11y version)?  Is there value in
> running this without a11y turned on?  

I don't think we need both. Regressions could be tracked to their source via landings. Personally I think we should be testing will ally on by default for tp5 at least on windows. It's part of the code base and deserves good test coverage.

> For reference tp5r above is much different than tp5n.  tp5n has removed all
> outside network calls and 404s.  It would be better to compare tp5n with and
> without a11y.

What use is tp5r? Diagnosing network related regressions?
I definitely think that it's valuable to test both with and without a11y enabled.

But honestly people regress Talos tests all the time and ignore the test results.  I've mostly lost my faith at how much gain we're having from running Talos altogether... :(
tp5r was the test before tp5n, the only difference is the pageset which was cleaned up in tp5n.  

We are looking to run tp5n on Windows 7 only (1 cycle vs 25 cycles) and collect xperf information, this will be in addition to the normal tp5n runs that occur on on all platforms per check.

Does it make sense to run this on linux and mac?  I am just trying to get a scope of work on this before writing patches.
(In reply to Joel Maher (:jmaher) from comment #10)
> Does it make sense to run this on linux and mac?  I am just trying to get a
> scope of work on this before writing patches.

Trevor, any comment here? I'm not familiar with accessibility use on these platforms.
(In reply to Ehsan Akhgari [:ehsan] from comment #9)
> I definitely think that it's valuable to test both with and without a11y
> enabled.
> 
> But honestly people regress Talos tests all the time and ignore the test
> results.  I've mostly lost my faith at how much gain we're having from
> running Talos altogether... :(

We're working on making Talos regression detection more reliable and detectable as well as having more information that developers can delve into: https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise

It is a long effort, but I hope we'll get there.  The alternative is pretty much not having performance testing, which just isn't an option.
> It's part of the code base and deserves good test coverage.

I agree we _should_ test with it, but we should also test without.  I would bet we have more users in the latter configuration than the former, and the performance characteristics can be ... quite different.  :(
(In reply to Jim Mathies [:jimm] from comment #11)
> (In reply to Joel Maher (:jmaher) from comment #10)
> > Does it make sense to run this on linux and mac?  I am just trying to get a
> > scope of work on this before writing patches.
> 
> Trevor, any comment here? I'm not familiar with accessibility use on these
> platforms.

As I understand it the gnome people are moving towards accessibility always being on, so I suspect it makes to do this for linux eventually, although maybe less urgent than windoes.

On mac I'm not aware of any general move towards accessibility being always on, but there seems to be a number of random programs that use the accessibility apis, however currently we only enable accessibility for voice over, not any of the other things that want it.  our impl of the mac accessibility api is also a bit lacking.  We may want test coverage here eventually but I suspect its the least urgent of these platforms.
(In reply to Jim Mathies [:jimm] from comment #11)
> (In reply to Joel Maher (:jmaher) from comment #10)
> > Does it make sense to run this on linux and mac?  I am just trying to get a
> > scope of work on this before writing patches.
> 
> Trevor, any comment here? I'm not familiar with accessibility use on these
> platforms.

As I understand it the gnome people are moving towards accessibility always being on, so I suspect it makes to do this for linux eventually, although maybe less urgent than windoes.

On mac I'm not aware of any general move towards accessibility being always on, but there seems to be a number of random programs that use the accessibility apis, however currently we only enable accessibility for voice over, not any of the other things that want it.  our impl of the mac accessibility api is also a bit lacking.  We may want test coverage here eventually, but I'd say its probably the least urgent platform.
Component: Talos → Mochitest
Whiteboard: feature=story c=testing u=developer p=tbd
Priority: -- → P2
Summary: Add accessibility coverage for Windows tp tests on mc → Story - Add accessibility coverage for Windows tp tests on mc
Whiteboard: feature=story c=testing u=developer p=tbd → feature=external
Summary: Story - Add accessibility coverage for Windows tp tests on mc → Add accessibility coverage for Windows tp tests on mc
You need to log in before you can comment on or make changes to this bug.