Closed Bug 572486 Opened 10 years ago Closed 9 years ago

Create a talos measure for accessibility performance

Categories

(Testing :: Talos, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: davidb, Assigned: davidb)

References

Details

Attachments

(4 files, 5 obsolete files)

I have started creating this measure locally, but after speaking with Alice I think it makes sense to file the bug now.

Two things to watch here (due to a11y):

1. Once our a11y engine is instantiated (required for the a11y talos measure), it will effect subsequent performance measures unless FF is restarted.
2. When this lands, we shouldn't run it against OSX (where we don't build with accessibility enabled by default).
How do I create a cvs diff against talos trunk? (I can't cvs add - no write access)
Attached patch WIP (obsolete) — Splinter Review
Alice please advise if this is looks okay. For the config, I copied dromaeo.. and called the tests ta11y.
Attachment #452713 - Flags: feedback?(anodelman)
Comment on attachment 452713 [details] [diff] [review]
WIP

>Index: page_load_test/a11y/a11y.manifest
>===================================================================
>RCS file: page_load_test/a11y/a11y.manifest
>diff -N page_load_test/a11y/a11y.manifest
>--- /dev/null	1 Jan 1970 00:00:00 -0000
>+++ page_load_test/a11y/a11y.manifest	21 Jun 2010 14:01:47 -0000
>@@ -0,0 +1,3 @@
>+static.html
>+dhtml.html
>+tablemutation.html

Note to self: I forgot the %'s here.
Attached patch patch 1 (obsolete) — Splinter Review
Attachment #452713 - Attachment is obsolete: true
Attachment #453101 - Flags: review?(anodelman)
Attachment #452713 - Flags: feedback?(anodelman)
I'm not seeing these tests actually cycle - do they need to be run against a particular build or with a particular pref set?

Also, I don't see your updates to sample.config in your patch.
Upon looking at the console it is dying in initAccessibility().
Right I didn't copy my sample config over from my standalone tree. I'll do that.

(In reply to comment #6)
> Upon looking at the console it is dying in initAccessibility().

Oh dear, what platform are you testing on? Note: Mac release builds are not expected to work.
Alice, please let me know how this goes. (Find me on IRC if you can 'davidb')
Attachment #453101 - Attachment is obsolete: true
Attachment #454724 - Flags: feedback?(anodelman)
Attachment #453101 - Flags: review?(anodelman)
Can you give me a list of branches/platforms that this should be run on?
Sure. I think for now we want only Windows and Linux trunk.
Testing on talos-r3-fed-002.
The initial manual run looked good, I'm going to put together the necessary buildbot/graph server patches so that we can do a full staging run.

Would this be okay only run on moz-central win/linux?

Are you interested in 64bit results at all?
Initial manual test run on linux:

i,page,median,mean,min,max,runs
0,static.html,1,1,1,14,14|1|1|1|1
1,dhtml.html,44,44,41,57,57|41|46|42|47
2,tablemutation.html,37.5,38.25,35,47,40|35|47|35|43

Do the results look in line with your expectations?  Everything is loading in under a second (as the measure is in ms)...
(In reply to comment #12)
> The initial manual run looked good, I'm going to put together the necessary
> buildbot/graph server patches so that we can do a full staging run.
> 
> Would this be okay only run on moz-central win/linux?

Yes that's exactly right thanks.

> Are you interested in 64bit results at all?

If there are no gotchas, yes please.
(In reply to comment #13)
> Initial manual test run on linux:
> 
> i,page,median,mean,min,max,runs
> 0,static.html,1,1,1,14,14|1|1|1|1
> 1,dhtml.html,44,44,41,57,57|41|46|42|47
> 2,tablemutation.html,37.5,38.25,35,47,40|35|47|35|43
> 
> Do the results look in line with your expectations?  Everything is loading in
> under a second (as the measure is in ms)...

I'm not sure. Here is what I get on my OS X debug accessibility build:

i,page,median,mean,min,max,runs
0,static.html,4,3.6666666666666665,3,5,5|3|4|4
1,dhtml.html,132,126.66666666666667,109,157,109|139|157|132
2,tablemutation.html,106,112,105,125,106|125|125|105
RETURN: ta11y: 55.0

I suppose the fact it is debug might be the difference.

The static.html test looks too quick doesn't it... do you recommend I make it more involved -- take longer?
Actually the static.html used to take a lot longer -- we've had some recent perf wins. I'll modify it.

Alice are you sure a11y was invoked in your manual tests (the build had accessibility enabled)?
How is accessibility enabled?  Is there a pref that needs setting?

I feel like I'm missing the whole test set up here...
(In reply to comment #17)
> How is accessibility enabled?  Is there a pref that needs setting?

Most build configurations default to building an accessibility enabled firefox. There is a pref to force accessibility off, but that defaults to false.

You can check a build with https://addons.mozilla.org/en-US/firefox/addon/2407/

> I feel like I'm missing the whole test set up here...

I'll be around all week 8-4 EST davidb on IRC :)
I wasn't clear actually.

To check if accessibility is available you can try evaluating this in the js console:
Components.classes["@mozilla.org/accessibleRetrieval;1"].
    getService(nsIAccessibleRetrieval);

To check if accessibility is available and 'activated' use:
Components.manager.QueryInterface(CI.nsIServiceManager)
              .isServiceInstantiatedByContractID(
                  "@mozilla.org/accessibilityService;1",
                  CI.nsISupports);

To ensure you are running the tests on a relevant build it should be enough to perform the first check.
Okay, I tried:

Components.classes["@mozilla.org/accessibleRetrieval;1"].
    getService(nsIAccessibleRetrieval);

And the nsIAccessibleRetrieval was undefined.  So that would be why my test results were so low.  I'm using a Firefox3.6 tinderbox build.  Is accessibility only on in the nightlies?  Only release builds?
I'm not familiar with the config details for the tinderbox builds but generally we don't build-in accessibility on OS X. If you test with a windows or linux build (perhaps a nightly), you should be good to go.
I've been working on linux exclusively after your first warning about osx... guess that I'll have to try some more builds until one works?

Do we build in accessibility on all branches?
Ugh paste error - sorry! Please try:
Components.classes["@mozilla.org/accessibleRetrieval;1"].getService(Components.interfaces.nsIAccessibleRetrieval);
And:
Components.manager.QueryInterface(Components.interfaces.nsIServiceManager).isServiceInstantiatedByContractID("@mozilla.org/accessibilityService;1",Components.interfaces.nsISupports);

To see if a11y is active.
I'm going to drop the static.html test since we nailed that one already in our performance work.
Updated patch. I added but didn't test the a11y init fail path.
Attachment #454724 - Attachment is obsolete: true
Attachment #459422 - Flags: review?(anodelman)
Attachment #454724 - Flags: feedback?(anodelman)
Looks to be running okay, but could you update the patch for sample.config to call the test 'a11y' instead of 'accessibility' - it'll fit on the waterfall better.
Attachment #460355 - Flags: review?(bhearsum)
(In reply to comment #27)
> Looks to be running okay, but could you update the patch for sample.config to
> call the test 'a11y' instead of 'accessibility' - it'll fit on the waterfall
> better.

Yep! Sorry for delay.
Attachment #459422 - Attachment is obsolete: true
Attachment #459422 - Flags: review?(anodelman)
Graph server db update applied to graphs-stage.
Attachment #460559 - Flags: review?(bhearsum) → review+
Comment on attachment 460355 [details] [diff] [review]
add a11y to talos buildbot-configs

Have you explicitly tested 1.9.1 and 1.9.2? I'd like to be sure that this works there prior to landing.

I think talos-pool is obsolete -- you can probably omit that one line change.

Looks fine otherwise.
Attachment #460355 - Flags: review?(bhearsum) → review+
Comment on attachment 460355 [details] [diff] [review]
add a11y to talos buildbot-configs

Flipping to r- based on this comment:

(In reply to comment #10)
> Sure. I think for now we want only Windows and Linux trunk.


Except, we'll want it for all branches except 1.9.1 and 1.9.2 since m-c project branches and try need to track trunk's configuration.
Attachment #460355 - Flags: review+ → review-
Graph server db update applied to production graphs.
Attachment #460355 - Attachment is obsolete: true
Attachment #460626 - Flags: review?(bhearsum) → review+
No longer blocks: 581098
Blocks: 581098
Comment on attachment 460559 [details] [diff] [review]
[checked in]add a11y to graph server

changeset:   315:4ba11ef8b1d3
Attachment #460559 - Attachment description: add a11y to graph server → [checked in]add a11y to graph server
Comment on attachment 460516 [details] [diff] [review]
[checked in]patch 1.3

Checking in sample.config;
/cvsroot/mozilla/testing/performance/talos/sample.config,v  <--  sample.config
new revision: 1.48; previous revision: 1.47
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.js,v
done
Checking in page_load_test/a11y/a11y.js;
/cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.js,v  <--  a11y.js
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.manifest,v
done
Checking in page_load_test/a11y/a11y.manifest;
/cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/a11y.manifest,v  <--  a11y.manifest
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/dhtml.html,v
done
Checking in page_load_test/a11y/dhtml.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/dhtml.html,v  <--  dhtml.html
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/tablemutation.html,v
done
Checking in page_load_test/a11y/tablemutation.html;
/cvsroot/mozilla/testing/performance/talos/page_load_test/a11y/tablemutation.html,v  <--  tablemutation.html
initial revision: 1.1
done
Attachment #460516 - Attachment description: patch 1.3 → [checked in]patch 1.3
Comment on attachment 460626 [details] [diff] [review]
[checked in]add a11y to talos buildbot-configs (take 2) exclude 1.9.1/1.9.2

changeset:   2766:603bfc3a61c8
Attachment #460626 - Attachment description: add a11y to talos buildbot-configs (take 2) exclude 1.9.1/1.9.2 → [checked in]add a11y to talos buildbot-configs (take 2) exclude 1.9.1/1.9.2
Bustage fix for missing graph server config in buildbot-configs:

changeset:   2767:ab93026d32e5
Already seeing successful green runs on mozilla-central.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Alice, thanks for your help and patience!
Don't forget to check 'Scrape' on the tinderbox pages after deploying things like this.
You need to log in before you can comment on or make changes to this bug.