Open
Bug 778493
Opened 12 years ago
Updated 2 years ago
Add accessibility coverage for Windows tp tests on mc
Categories
(Testing :: Mochitest, defect, P2)
Tracking
(Not tracked)
NEW
People
(Reporter: jimm, Unassigned)
References
Details
(Whiteboard: feature=external)
On Windows touch enabled devices accessibility is loaded by default. We currently don't run talos tp tests though with accessibility. Currently the number of machines out there with touch input is low but that will likely change with the release of win8. I did quick test on try server with a patch that enabled a11y by forcing the load of the root accessible object, the difference in tp is noticeable - accessibility: tp5r_paint: 368.73 tp5r_modlistbytes_paint: 60.6MB tp5r_%cpu_paint: 76.42 tp5r_content_rss_paint: 76.4B tp5r_main_rss_paint: 76.4B tp5r_pbytes_paint: 165.5MB tp5r_responsiveness_paint: 518.0 tp5r_shutdown_paint: 1356.0 without: tp5n_paint: 275.03 tp5n_modlistbytes_paint: 59.5MB tp5n_%cpu_paint: 71.81 tp5n_content_rss_paint: 71.7B tp5n_main_rss_paint: 71.7B tp5n_pbytes_paint: 128.4MB tp5n_responsiveness_paint: 871.0 tp5n_shutdown_paint: 1318.0 Without test coverage here we can't work toward lower drag from the a11y code, so I'd like to propose we either run all our tp tests with a11y forced on, or run a new set of tp tests with it on. Note I didn't see any appreciable difference in other talos tests in my try run. The results are here - https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884 We can control the forcing of a11y on through a pref.
Comment 1•12 years ago
|
||
(In reply to Jim Mathies [:jimm] from comment #0) > On Windows touch enabled devices accessibility is loaded by default. We > currently don't run talos tp tests though with accessibility. Currently the > number of machines out there with touch input is low but that will likely > change with the release of win8. > > I did quick test on try server with a patch that enabled a11y by forcing the > load of the root accessible object, the difference in tp is noticeable - > > accessibility: > > tp5r_paint: 368.73 > tp5r_modlistbytes_paint: 60.6MB > tp5r_%cpu_paint: 76.42 > tp5r_content_rss_paint: 76.4B > tp5r_main_rss_paint: 76.4B > tp5r_pbytes_paint: 165.5MB > tp5r_responsiveness_paint: 518.0 > tp5r_shutdown_paint: 1356.0 > > without: > > tp5n_paint: 275.03 > tp5n_modlistbytes_paint: 59.5MB > tp5n_%cpu_paint: 71.81 > tp5n_content_rss_paint: 71.7B > tp5n_main_rss_paint: 71.7B > tp5n_pbytes_paint: 128.4MB > tp5n_responsiveness_paint: 871.0 > tp5n_shutdown_paint: 1318.0 that's noticable, but I'm actually sort of happy to see it isn't worse. I'm also a little suprised those are the only tests effected. > Without test coverage here we can't work toward lower drag from the a11y > code, so I'd like to propose we either run all our tp tests with a11y forced > on, or run a new set of tp tests with it on. this seems like a good idea, but I'm not sure exactly which approach is best. > We can control the forcing of a11y on through a pref. yes, or from chrome code we can get nsIAccessibleRetrieval
Comment 2•12 years ago
|
||
> Note I didn't see any appreciable difference in other talos tests in my try
> run. The results are here -
>
> https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884
not really related, but we might want to atleast check correctness tests aren't effected too.
Reporter | ||
Comment 3•12 years ago
|
||
(In reply to Trevor Saunders (:tbsaunde) from comment #2) > > Note I didn't see any appreciable difference in other talos tests in my try > > run. The results are here - > > > > https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884 > > not really related, but we might want to atleast check correctness tests > aren't effected too. By correctness tests are you referring to reftests?
Reporter | ||
Comment 4•12 years ago
|
||
Here's a talos compare between the two: http://perf.snarkfest.net/compare-talos/index.html?oldRevs=6f2382473be3&newRev=188765a0088a&submit=true
Comment 5•12 years ago
|
||
(In reply to Jim Mathies [:jimm] from comment #3) > (In reply to Trevor Saunders (:tbsaunde) from comment #2) > > > Note I didn't see any appreciable difference in other talos tests in my try > > > run. The results are here - > > > > > > https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884 > > > > not really related, but we might want to atleast check correctness tests > > aren't effected too. > > By correctness tests are you referring to reftests? I'd think only mochitests and reftests could reasonably be effected.
Comment 6•12 years ago
|
||
(In reply to Trevor Saunders (:tbsaunde) from comment #5) > (In reply to Jim Mathies [:jimm] from comment #3) > > (In reply to Trevor Saunders (:tbsaunde) from comment #2) > > > > Note I didn't see any appreciable difference in other talos tests in my try > > > > run. The results are here - > > > > > > > > https://tbpl.mozilla.org/?noignore=0&tree=Try&rev=f4ec51043884 > > > > > > not really related, but we might want to atleast check correctness tests > > > aren't effected too. > > > > By correctness tests are you referring to reftests? > I'd think only mochitests and reftests could reasonably be effected. so, yes, reftests and mochitests was what I meant
Comment 7•12 years ago
|
||
would you want to replace tp5n with tp5a (a11y version)? Is there value in running this without a11y turned on? For reference tp5r above is much different than tp5n. tp5n has removed all outside network calls and 404s. It would be better to compare tp5n with and without a11y.
Reporter | ||
Comment 8•12 years ago
|
||
(In reply to Joel Maher (:jmaher) from comment #7) > would you want to replace tp5n with tp5a (a11y version)? Is there value in > running this without a11y turned on? I don't think we need both. Regressions could be tracked to their source via landings. Personally I think we should be testing will ally on by default for tp5 at least on windows. It's part of the code base and deserves good test coverage. > For reference tp5r above is much different than tp5n. tp5n has removed all > outside network calls and 404s. It would be better to compare tp5n with and > without a11y. What use is tp5r? Diagnosing network related regressions?
Comment 9•12 years ago
|
||
I definitely think that it's valuable to test both with and without a11y enabled. But honestly people regress Talos tests all the time and ignore the test results. I've mostly lost my faith at how much gain we're having from running Talos altogether... :(
Comment 10•12 years ago
|
||
tp5r was the test before tp5n, the only difference is the pageset which was cleaned up in tp5n. We are looking to run tp5n on Windows 7 only (1 cycle vs 25 cycles) and collect xperf information, this will be in addition to the normal tp5n runs that occur on on all platforms per check. Does it make sense to run this on linux and mac? I am just trying to get a scope of work on this before writing patches.
Reporter | ||
Comment 11•12 years ago
|
||
(In reply to Joel Maher (:jmaher) from comment #10) > Does it make sense to run this on linux and mac? I am just trying to get a > scope of work on this before writing patches. Trevor, any comment here? I'm not familiar with accessibility use on these platforms.
Comment 12•12 years ago
|
||
(In reply to Ehsan Akhgari [:ehsan] from comment #9) > I definitely think that it's valuable to test both with and without a11y > enabled. > > But honestly people regress Talos tests all the time and ignore the test > results. I've mostly lost my faith at how much gain we're having from > running Talos altogether... :( We're working on making Talos regression detection more reliable and detectable as well as having more information that developers can delve into: https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise It is a long effort, but I hope we'll get there. The alternative is pretty much not having performance testing, which just isn't an option.
Comment 13•12 years ago
|
||
> It's part of the code base and deserves good test coverage.
I agree we _should_ test with it, but we should also test without. I would bet we have more users in the latter configuration than the former, and the performance characteristics can be ... quite different. :(
Comment 14•12 years ago
|
||
(In reply to Jim Mathies [:jimm] from comment #11) > (In reply to Joel Maher (:jmaher) from comment #10) > > Does it make sense to run this on linux and mac? I am just trying to get a > > scope of work on this before writing patches. > > Trevor, any comment here? I'm not familiar with accessibility use on these > platforms. As I understand it the gnome people are moving towards accessibility always being on, so I suspect it makes to do this for linux eventually, although maybe less urgent than windoes. On mac I'm not aware of any general move towards accessibility being always on, but there seems to be a number of random programs that use the accessibility apis, however currently we only enable accessibility for voice over, not any of the other things that want it. our impl of the mac accessibility api is also a bit lacking. We may want test coverage here eventually but I suspect its the least urgent of these platforms.
Comment 15•12 years ago
|
||
(In reply to Jim Mathies [:jimm] from comment #11) > (In reply to Joel Maher (:jmaher) from comment #10) > > Does it make sense to run this on linux and mac? I am just trying to get a > > scope of work on this before writing patches. > > Trevor, any comment here? I'm not familiar with accessibility use on these > platforms. As I understand it the gnome people are moving towards accessibility always being on, so I suspect it makes to do this for linux eventually, although maybe less urgent than windoes. On mac I'm not aware of any general move towards accessibility being always on, but there seems to be a number of random programs that use the accessibility apis, however currently we only enable accessibility for voice over, not any of the other things that want it. our impl of the mac accessibility api is also a bit lacking. We may want test coverage here eventually, but I'd say its probably the least urgent platform.
Reporter | ||
Updated•12 years ago
|
Blocks: metro-testing
Reporter | ||
Updated•11 years ago
|
Component: Talos → Mochitest
Updated•11 years ago
|
Whiteboard: feature=story c=testing u=developer p=tbd
Updated•11 years ago
|
Priority: -- → P2
Summary: Add accessibility coverage for Windows tp tests on mc → Story - Add accessibility coverage for Windows tp tests on mc
Updated•11 years ago
|
Whiteboard: feature=story c=testing u=developer p=tbd → feature=external
Updated•11 years ago
|
Summary: Story - Add accessibility coverage for Windows tp tests on mc → Add accessibility coverage for Windows tp tests on mc
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•