Closed Bug 424195 Opened 16 years ago Closed 16 years ago

Start running a11y Mochitests regularly.

Categories

(Release Engineering :: General, defect, P3)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: MarcoZ, Assigned: lsblakk)

References

Details

Attachments

(10 files, 1 obsolete file)

mozconfig with a11y enabled
524 bytes, text/plain
Details
mac errors log
9.90 KB, text/plain
Details
linux errors log
9.42 KB, text/plain
Details
[checked in] a11y buildbot configs staging
23.97 KB, patch
coop
: review+
Details | Diff | Splinter Review
[backed out] master.cfg changes for build/buildbot-configs/mozilla2-unittest
2.18 KB, patch
coop
: review+
Details | Diff | Splinter Review
Add a11y tests to steps.py
2.67 KB, patch
Details | Diff | Splinter Review
Linux a11y test fail output from stdio on M-C staging
11.33 KB, application/text
Details
Win32 a11y test fail output from stdio on M-C staging
73.90 KB, application/text
Details
memory reference error
52.81 KB, image/png
Details
re-enable a11y for linux, win32 on both mozilla-central and mozilla-1.9.1
3.53 KB, patch
coop
: review+
Details | Diff | Splinter Review 
Requesting that the a11y Mochitests be run regularly to automatically catch possible regressions. There aren't that many yet, but those we have are already good for several situations. The parameter for runtests.py and runtests.pl is --a11y .
re-classifying and prioritizing. OK to take this.
Component: Release Engineering: Talos → Release Engineering: Future
Priority: -- → P3
Marco: I ran this on a local build yesterday just to see if it's working. The harness came up as it should, but there were no tests showing. A couple of questions:

1) Do I need to turn on any special configure options?

2) Are there any tests checked in?

thanks!
Assignee: nobody → rcampbell
Priority: P3 → P2
(In reply to comment #2)
> 1) Do I need to turn on any special configure options?

You need to run the runtests.pl or runtests.py with the --a11y switch. Then, the "Accessibility testharness" page comes up. When you click the "Run tests" button, you usually get 479 or so passes, a bunch of todo's, and hopefully no failures. :-)

> 2) Are there any tests checked in?

Most definitely! They are stored in mozilla/accessible/tests/mochitest.

One thing that just came to my mind: Are you trying this on Mac? If so, that may be why you're not able to see/run them.

1. Mac accessibility is practically non-existent (for supporting the Universal Access APIs, that is), so these tests are currently only making sense to run on Win and Linux.
2. The build system may not pick the a11y testfiles up because of that very reason.
I am indeed running on a Mac.

I'll try rerunning this under linux and windows. I guess it's safe to say that we won't need to add these tests to the Mac unittest boxes then?
(In reply to comment #5)
> I guess it's safe to say that
> we won't need to add these tests to the Mac unittest boxes then?

Right! Until we have full Mac accessibility, which I currently see for Mozilla2 rather than Gecko 1.9.1, we don't need to clutter up the Mac unittest boxes with these.
Since mochitest make tests for Gecko stuff code only then it make sense to run them on Mac too. Just build should be built with enabled accessibility. Also, I think it's important to make builds with enabled accessibility to make sure we don't broke code on Mac.
(In reply to comment #7)
> Since mochitest make tests for Gecko stuff code only then it make sense to run
> them on Mac too. Just build should be built with enabled accessibility. Also, I
> think it's important to make builds with enabled accessibility to make sure we
> don't broke code on Mac.

Alex, thanks for the heads-up! Is there a special option that should be set that is not required on other platforms? For example, on Linux and Win, accessibility comes with the standard build options, I don't need to enable it explicitly. Could you give Rob some pointers?
Yes, that's option is ac_add_options --enable-accessibility. I use it when I build firefox on mac.
good advice. Thanks for the help, Alex and Marco!
with the attached .mozconfig file, I ran mac and linux.

The mac run completed with:
*** 874 INFO Passed: 705
*** 875 INFO Failed: 60
*** 876 INFO Todo:   78

the linux run completed with:
*** 875 INFO Passed: 707
*** 876 INFO Failed: 60
*** 877 INFO Todo:   77

Do these numbers look right or am I missing something? Failures seems high.

See other attachments for mac and linux error logs.
(In reply to comment #11)
> Do these numbers look right or am I missing something? Failures seems high.

The last 9 errors (in tableinsane2 and tableinsane4) have already been filed as bug 437980.

The Mac error at the top is very strange. It almost appears as if that particular property is not exposed to JS on Mac. Did this one come up everytime?

The top Linux problems in test_bug368835.xul may be fixed by bug 427841, but I am not sure. Also, this is something I've seen come and go, and it almost never happens if the test is run alone and not as part of the full run.

The wrong X and Y coordinates for images are very bizarre. If I run that test stand-alone, the images appear at the spots I specify in the mochitest. If the test runs within all others, the images appear in a different place. Any recommended course of action other than leaving these tests out?

(In reply to comment #14)
> The wrong X and Y coordinates for images are very bizarre. If I run that test
> stand-alone, the images appear at the spots I specify in the mochitest. If the
> test runs within all others, the images appear in a different place. Any
> recommended course of action other than leaving these tests out?

Filed bug 439057 for this issue.
Depends on: 441974
first crack at automation. Successful run on staging on mac and windows.
Attachment #326991 - Flags: review?(ccooper)
Comment on attachment 326991 [details] [diff] [review]
[checked in] a11y buildbot configs staging

You import copy, but then don't seem to use it. 

r+ otherwise.
Attachment #326991 - Flags: review?(ccooper) → review+
Comment on attachment 326991 [details] [diff] [review]
[checked in] a11y buildbot configs staging

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/testing/unittest-stage/master.cfg,v  <--  master.cfg
new revision: 1.11; previous revision: 1.10
done
Checking in mozbuild.py;
/cvsroot/mozilla/tools/buildbot-configs/testing/unittest-stage/mozbuild.py,v  <--  mozbuild.py
new revision: 1.3; previous revision: 1.2
done
Attachment #326991 - Attachment description: a11y buildbot configs staging → [checked in] a11y buildbot configs staging
Marco, Rob:

1) What o.s. are we running these tests on? Comment#4 talks about not being on mac, but comment#11 talks about running on mac. 

2) If we disable the Failed+ToDo tests, could we start running the Passing tests in production?  
(In reply to comment #20)
> 1) What o.s. are we running these tests on? Comment#4 talks about not being on
> mac, but comment#11 talks about running on mac. 

I originally assumed that the accessibility code was not running at all on the Mac. But I was wrong about that: The platform-independent code is executed once someone asks for accessibility on the Mac, or turns it on as described above by Surkov. So the platform independent Mochitests (which are currently all of them) should definitely also run on Mac.

> 2) If we disable the Failed+ToDo tests, could we start running the Passing
> tests in production?  

I think so. Note however, that the more exciting stuff is taking place in mozilla-central, not on the 1.9.0 branch. So, we should get them running on something running off of mozilla-central. That's where testing needs to catch regressions, and if something is good or has baked there for a while, it can then be backported to branch if needed.
(In reply to comment #21)
> (In reply to comment #20)
> > 2) If we disable the Failed+ToDo tests, could we start running the Passing
> > tests in production?  
> 
> I think so. Note however, that the more exciting stuff is taking place in
> mozilla-central, not on the 1.9.0 branch. So, we should get them running on
> something running off of mozilla-central. That's where testing needs to catch
> regressions, and if something is good or has baked there for a while, it can
> then be backported to branch if needed.
> 
Marco, can you disable those failing a11y tests? We'd like to enable the tests that *do* pass on production asap.
Assignee: rcampbell → lukasblakk
(In reply to comment #22)
> Marco, can you disable those failing a11y tests? We'd like to enable the tests
> that *do* pass on production asap.

I have a fix for bug 439057 in the pipeline that will make those tests pass.
bug 437980 we don't have a patch yet, so I'd need to disable all these table_indexes tests (the whole file), since the tests are part of a loop that also has passing ones.
As for general performance: bug 441974 may also be visible on production or staging machines. On my system, it causes some tests to fail randomly when running alltogether, but passing whenever I run the file in question stand-alone. If you or Lucas have any idea on that, please comment on that bug. Clint said he'd also take a look.
hi Marco;

1) Any ETA on when the patch can be landed, and other failing tests disabled? We'd really like to get this into production for 1.9. 

2) I note the patch above is for cvs. Do you have a similar patch incoming for these tests on hg?
(In reply to comment #24)
> hi Marco;
> 
> 1) Any ETA on when the patch can be landed, and other failing tests disabled?
> We'd really like to get this into production for 1.9. 

Patch for bug 439057 landed a few days ago on HG, fixing those big numbers of failing tests. Leaves the table_indexes.html to be disabled, and the other bug 441974 to work out why some tests fail randomly. I'm desperately waiting on feedback from Clint and Sayrer on that one.

> 2) I note the patch above is for cvs. Do you have a similar patch incoming for
> these tests on hg?

Rob, can you speak more to that?
However, I didn't find the mozbuild.py file at all in this repo. Would appreciate some help on this!
Attachment #331052 - Flags: review?
Attachment #331052 - Flags: review? → review?(ccooper)
http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest-stage/mozbuild.py


However, there is a patch waiting to come in here - https://bugzilla.mozilla.org/show_bug.cgi?id=440164  that you should look at before making changes to the current revision of mozbuild.py

I'll try to get that checked in today.


Comment on attachment 331052 [details] [diff] [review]
[backed out] master.cfg changes for build/buildbot-configs/mozilla2-unittest

Looks fine, assuming the A11y buildsteps are defined somewhere else (mozbuild.py, buildbotcustom, ...). 

Is there another patch forthcoming for those?
Attachment #331052 - Flags: review?(ccooper) → review+
(In reply to comment #28)
> Looks fine, assuming the A11y buildsteps are defined somewhere else
> (mozbuild.py, buildbotcustom, ...). 
> 
> Is there another patch forthcoming for those?

Do you have any hints of where I could look for this? I synced with http://hg.mozilla.org/build/buildbot-configs, but didn't find a suitabe mozilla2/Firefox file where I could put that second half of the original patch I converted. config.py is present, but it contains totally different stuff than the ones that were in CVS. So any hints as to where these other steps need to be defined would be greatly appreciated!
(In reply to comment #27)
> http://mxr.mozilla.org/mozilla/source/tools/buildbot-configs/testing/unittest-stage/mozbuild.py

I'm actually talking Mercurial here, not CVS trunk. Joduinn asked whether we could get the tests running on the Mozilla-Central tinderboxen.
For the mercurial staging and production, we use buildbotcustom steps.py as seen here:

http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/unittest/steps.py

(we still check out our configs from cvs for now, there is no repo in hg yet for buildbot)

Steps.py replaces mozbuild.py
Attached patch Add a11y tests to steps.py (obsolete) — — Splinter Review 
Ok, this adds the steps to steps.py which is what buildbotcustom looks to in the moz2 setup.

I'll request a win32 box for moz2 staging, for now we can try this on the linux box that's already running on staging.
Attachment #331741 - Flags: review?(ccooper)
Attachment #331741 - Flags: review?(ccooper) → review+
steps.py got updated by bug 448416, so this is a new patch to work with the updated version.
Attachment #331741 - Attachment is obsolete: true
Attachment #332395 - Flags: review+
Attachment #332395 - Flags: review+
Checking in tools/buildbotcustom/unittest/steps.py;
/cvsroot/mozilla/tools/buildbotcustom/unittest/steps.py,v  <--  steps.py
new revision: 1.12; previous revision: 1.11
done

Checked in on Lukas' behalf. Thanks!
Keywords: checkin-needed
Pushed master.cfg changes to build/buildbot-configs in changeset:
http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/4af357980258
 (In reply to comment #35)
> Pushed master.cfg changes to build/buildbot-configs in changeset:
> http://hg.mozilla.org/build/buildbot-configs/index.cgi/rev/4af357980258

So, the big block of failing tests are disabled, bug#441974 filed to track other intermittent test failures and now this patch is landed.

Whats left to do in this bug before we have a11y tests running on mozilla-central?
I don't know if we're still waiting for a checkin or something, but the a11y tests are still throwing warnings on both staging 1.9 and staging m-c - until I see some green, these can't move to production.


marco, ted: until bug#441974 is resolved, can we just mark those specific tests as "expected fail"?

At least that would get the remaining passing tests to show green, and we could then enable those into production. Right now, we have nothing in production. :-(
marco, ted: any update? 

This is a Q3 goal, and we're only weeks away from end of quarter.
Depends on: 454654
Attachment #331052 - Attachment description: master.cfg changes for build/buildbot-configs/mozilla2-unittest → [backed out] master.cfg changes for build/buildbot-configs/mozilla2-unittest
Comment on attachment 331052 [details] [diff] [review]
[backed out] master.cfg changes for build/buildbot-configs/mozilla2-unittest

I backed this change out. It never made it into production and is blocking other changes we want to take. We can reland and deploy (in close succession!) when the time is right.
So I tried turning on the a11y tests in moz2 unittests.

Here is what the results were:

Win32 (7 tests fail consistently):

*** 851 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No focused state for element with ID aria-link2.
*** 919 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area1 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 935 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area2 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 980 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area4 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 998 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area5 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 1017 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area6 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 1048 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area7 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"

On Linux (22 fails consistent):

79:*** 32 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_bug420863.html | Error thrown during test: uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numActions]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 :: line 97"  data: no] - got 0, expected 1
153:*** 112 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item1.1 - got "", expected "1"
154:*** 113 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item1.1 - got "", expected "1"
156:*** 115 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item1.2 - got "", expected "1"
157:*** 116 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item1.2 - got "", expected "3"
159:*** 118 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item1.4 - got "", expected "2"
160:*** 119 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item1.4 - got "", expected "3"
162:*** 121 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset on menu_item2 - got "", expected "3"
163:*** 122 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize on menu_item2 - got "", expected "3"
419:*** 385 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImage! - got 587, expected 588
441:*** 407 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageWithAlt! - got 713, expected 714
463:*** 429 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageWithTitle! - got 839, expected 840
485:*** 451 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageEmptyAlt! - got 965, expected 966
507:*** 473 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong screen y coordinate for linkedImageEmptyAltAndTitle! - got 1091, expected 1092
563:*** 530 ERROR TEST-UNEXPECTED-PASS | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleTable_1.html | columnHeader should not throw
876:*** 850 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No focused state for element with ID aria-link.
942:*** 920 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area1 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
958:*** 936 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area2 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1003:*** 981 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area4 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1021:*** 999 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area5 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1040:*** 1018 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area6 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1071:*** 1049 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute 'background-color 'has wrong value. Getting default attributes for area7 - got "rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
Here is the output of errors on 1.9 staging:

Win32 pgo build (11 fails):

95:*** 35 ERROR FAIL | Error thrown during test: uncaught exception: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numActions]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 :: line 97"  data: no] | got 0, expected 1 | chrome://mochikit/content/a11y/accessible/test_bug420863.html
683:*** 638 ERROR FAIL | Wrong name for internal frame! | got null, expected "" | chrome://mochikit/content/a11y/accessible/test_nsOuterDocAccessible.html
807:*** 763 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 0 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
811:*** 767 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 1 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
815:*** 771 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 2 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
835:*** 791 ERROR FAIL | tableinsane2: cell index from object attributes of cell accessible isn't corrent. | got 8, expected 9 | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
844:*** 800 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 3 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
848:*** 804 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 4 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
867:*** 823 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 0 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
871:*** 827 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 1 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
875:*** 831 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 2 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html

Win32 Build (only 10 error fails):

685:*** 638 ERROR FAIL | Wrong name for internal frame! | got null, expected "" | chrome://mochikit/content/a11y/accessible/test_nsOuterDocAccessible.html
809:*** 763 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 0 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
813:*** 767 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 1 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
817:*** 771 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 2 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
837:*** 791 ERROR FAIL | tableinsane2: cell index from object attributes of cell accessible isn't corrent. | got 8, expected 9 | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
846:*** 800 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 3 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
850:*** 804 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 4 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
869:*** 823 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 0 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
873:*** 827 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 1 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
877:*** 831 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 2 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html

Linux (24 error fail):

60:*** 12 ERROR FAIL | Wrong 'startrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got 5, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
61:*** 13 ERROR FAIL | Wrong 'endrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got 5, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
62:*** 14 ERROR FAIL | Wrong 'startcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got null, expected 0 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
63:*** 15 ERROR FAIL | Wrong 'endcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateColumn | got null, expected 0 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
64:*** 16 ERROR FAIL | Wrong 'startrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected 1 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
65:*** 17 ERROR FAIL | Wrong 'endrow' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected 1 | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
66:*** 18 ERROR FAIL | Wrong 'startcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
67:*** 19 ERROR FAIL | Wrong 'endcolumn' of 'treeInvalidated' event on nsITreeBoxObject::invalidateRow | got 0, expected null | chrome://mochikit/content/a11y/accessible/test_bug368835.xul
251:*** 211 ERROR FAIL | Wrong screen y coordinate for linkedImage! | got 560, expected 561 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html
273:*** 233 ERROR FAIL | Wrong screen y coordinate for linkedImageWithAlt! | got 686, expected 687 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html
295:*** 255 ERROR FAIL | Wrong screen y coordinate for linkedImageWithTitle! | got 812, expected 813 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html
317:*** 277 ERROR FAIL | Wrong screen y coordinate for linkedImageEmptyAlt! | got 938, expected 939 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html
339:,10001:0*** 299 ERROR FAIL | Wrong screen y coordinate for linkedImageEmptyAltAndTitle! | got 1064, expected 1065 | chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html
671:*** 639 ERROR FAIL | Wrong name for internal frame! | got null, expected "" | chrome://mochikit/content/a11y/accessible/test_nsOuterDocAccessible.html
795:*** 764 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 0 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
799:*** 768 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 1 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
803:*** 772 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 1, column = 2 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
823:*** 792 ERROR FAIL | ta,170:0bleinsane2: cell index from object attributes of cell accessible isn't corrent. | got 8, expected 9 | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
832:*** 801 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 3 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
836:*** 805 ERROR FAIL | tableinsane2: Can't get cell accessible at row = 4, column = 4 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
855:*** 824 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 0 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
859:*** 828 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 1 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
863:*** 832 ERROR FAIL | tableinsane4: Can't get cell accessible at row = 1, column = 2 |  | chrome://mochikit/content/a11y/accessible/test_table_indexes.html
All disabled/protected. Please try another run, thanks!
Thank you Marco, looking good on 1.9 - I see at least one green run on Linux, and Win32 (non-pgo) - will keep an eye on it while we wait for VMs to put on moz2 staging.  

Sorry for the delay, I hope we'll have this done within the week.
Ok, so I ran the a11y step on qm-moz2-unittest01 slaves and the results so far are:

Linux - green!
Win32 - http://pastebin.mozilla.org/545432

Marco - what do you think of the leaks on windows?  Any ideas what could be causing this?
Also - for further update, the 1.9 accessibility tests have all passed successfully at least once.  I believe we can go ahead and enable these on production 1.9 later this week.
(In reply to comment #46)
> Ok, so I ran the a11y step on qm-moz2-unittest01 slaves and the results so far
> are:
> 
> Linux - green!
> Win32 - http://pastebin.mozilla.org/545432
> 
> Marco - what do you think of the leaks on windows?  Any ideas what could be
> causing this?

I don't see any leaks there. Did you miss them in your paste, or am I missing something?
Sorry - here is the leak info from a build where the only orange was on the a11y test step:

nsTraceRefcntImpl::DumpStatistics: 876 entries
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 15868 bytes during test execution (should have leaked no more than 0 bytes)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of AtomImpl with size 16 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of ThreadShutdownObserver with size 12 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of imgLoader with size 24 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of mozStorageConnection with size 72 bytes each (144 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of mozStorageFunctionGetUnreversedHost with size 8 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of mozStorageService with size 24 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 48 instances of mozStorageStatement with size 40 bytes each (1920 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsAccessNode with size 20 bytes each (260 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsAccessible with size 72 bytes each (936 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsBaseAppShell with size 60 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsBaseURLParser with size 8 bytes each (16 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsBasicEncoder with size 12 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsCategoryManager with size 76 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsCollation with size 8 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsCollationWin with size 16 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsCommandManager with size 56 bytes each (112 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDOMStorage with size 64 bytes each (128 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDSURIContentListener with size 32 bytes each (64 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDateTimeFormatWin with size 36 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDefaultURIFixup with size 24 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDeque with size 52 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocLoader with size 144 bytes each (288 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocShell with size 448 bytes each (896 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocShell::InterfaceRequestorProxy with size 12 bytes each (24 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsDocumentCharsetInfo with size 20 bytes each (40 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsDownloadManager with size 56 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsHTMLImageAccessible with size 164 bytes each (2132 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 13 instances of nsHTMLImageAccessibleWrap with size 172 bytes each (2236 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsLayoutHistoryState with size 48 bytes each (96 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsLoadGroup with size 4 bytes each (8 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 3 instances of nsLocalFile with size 88 bytes each (264 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsLocale with size 16 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsNavBookmarks with size 192 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsNavHistory with size 808 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsObserverService with size 44 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsPersistentProperties with size 76 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsPrefBranch with size 52 bytes each (104 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsPrefService with size 36 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 4 instances of nsRect with size 16 bytes each (64 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsRunnable with size 8 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsSHEntry with size 140 bytes each (280 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 6 instances of nsStandardURL with size 172 bytes each (1032 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 193 instances of nsStringBuffer with size 8 bytes each (1544 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsStringBundle with size 32 bytes each (64 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsSystemPrincipal with size 32 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 4 instances of nsTArray_base with size 4 bytes each (16 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsThread with size 68 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsThreadPool with size 56 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsTimerImpl with size 48 bytes each (96 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 66 instances of nsVoidArray with size 4 bytes each (264 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 6 instances of nsWeakReference with size 12 bytes each (72 bytes total)
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsWebBrowserFind with size 48 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 1 instance of nsWebNavigationInfo with size 16 bytes
TEST-UNEXPECTED-FAIL | runtests-leaks | leaked 2 instances of nsWebShell with size 472 bytes each (944 bytes total)
 started: 2008-10-06 09:32:20.082000
finished: 2008-10-06 09:32:40.762000
program finished with exit code 0
So there's finally a pretty solid moz2-unittest staging setup now, and the a11y tests are being run on them. 

Right now, the linux tests are failing 37 tests and the win32 tests are throwing timeout exceptions.

Someone want to take a look?  

These slaves are reporting to Tinderbox tree=UnitTest

Linux:
moz2-linux-slave09
moz2-linux-slave10

Win32:
moz2-win32-slave09
moz2-win32-slave10
Just checked in again - one of the linux boxes is experiencing some pretty erratic results in the a11y tests, between 16 and 60 fails depending on the build, while the other is holding steady at 36 fails (and one instance of 37 fails).
(In reply to comment #50)
> So there's finally a pretty solid moz2-unittest staging setup now, and the a11y
> tests are being run on them. 
> Right now, the linux tests are failing 37 tests and the win32 tests are
> throwing timeout exceptions.
> Someone want to take a look?  

Marco, Ted: 

Who can best look at these failing a11y tests? The linux test *used* to pass green, so we feel their setup is good. The win32 timeout exception is recent, without us changing anything, so we're not sure whats causing that - any ideas?
The results of these test (logs, etc.) can be found at:

http://tinderbox.mozilla.org/showbuilds.cgi?tree=UnitTest
Depends on: 461190
Just did a clobber on the two win32 boxes. 

Is someone able to look at the exception that is being caused in a11y on win32?  Also, there are 22 tests failing in the Linux run, so can those be disabled for now so we can attempt to get this on production?
Marco or Surkov-  Can one of you disable those 22 tests so we can make progress this this bug (deploying to production) and resolving those failures in parallel?  Thx!
Lukas, can you send me just the failing lines of those logs for both Windows and Linux? I am not able to reproduce these 22 failures you are seeing, for whatever reason, and I am also having trouble reading those Tinderbox pages.

So it would be very helpful if you could get me the logs so I can disable the tests for now.

This is all mozilla-central, right?
(In reply to comment #56)
> Lukas, can you send me just the failing lines of those logs for both Windows
> and Linux? I am not able to reproduce these 22 failures you are seeing, for
> whatever reason, and I am also having trouble reading those Tinderbox pages.
Are you able to reproduce the problem if you run the entire suite? 



> So it would be very helpful if you could get me the logs so I can disable the
> tests for now.
Thanks Marco, it would be really great to deal with this, and finally get those suites running in production!

(In reply to comment #54)
> Is someone able to look at the exception that is being caused in a11y on win32?
Any ideas about these win32 exceptions?
This is what the most recent Linux test fails reported.

Marco, I hope this is what you need to be able to disable certain Linux tests.
and here's the windows output from the most recent a11y fails.
Looking at these logs, I see very strange things output already on the first two lines of each file:

1. On Windows, a global array variable is claimed not to be defined, even though its definition is clearly visible a few lines above the first use. In a different file, a constant is claimed to be used (and undefined) that appears nowhere within that file.
2. On Linux, the nsISupports interface is claimed not to be defined. Huh? All NSIAccessible* interfaces inherit from nsISupports and work fine.

I am not aware that anything within the a11y module could have caused this dramatic kindof failure.

In chatting with Lukas, we decided to run the a11y tests manually to see if the output is different.

Question: Did any other unit test parts have this dramatic kind of failure recently, for example after turning on JIT, and were there specific steps that needed to be taken when this happened?
So I ran these manually on a windows build.  Didn't run the other test suites first.  

The a11y suite runs, up to chrome://mochikit/content/a11y/accessible/test_textboxes.xul

it reports passing 932, failing 80 and todo 78 (if --close-when-done is included in the command)

it reports passing 932, failing 135 and todo 78 (if no --close-when-done)  and then when you close the window, you get the leak output as well as a memory reference error. (see attached)
So on the linux build, running the a11y tests also stops short at the test_textboxes.xul test, Passed 804, Failed 76, Todo 77.
Per irc, marco landed patch to disable the hanging "test_testboxes" suite, which should get us past these known failing tests. If this test run finds no other problems, then we'd enable this "as is" into production immediately.

bug#465755 filed to track debugging those specific test/code failures. We'll continue tracking infrastructure setup in this bug.
Marco: 

On http://tinderbox.mozilla.org/showbuilds.cgi?tree=UnitTest, I just noticed 

a11y 518/0/79

...is that success? :-) 

I also noticed only win32 unittest running... where did the linux unittests go?
John, I still have two patched files waiting to be committed. With all the branching, tagging etc. going on, I decided to wait until the dust cleared. This didn't seem to be the case throughout all Friday, so I didn't make a move yet.
(In reply to comment #66)
> John, I still have two patched files waiting to be committed. With all the
> branching, tagging etc. going on, I decided to wait until the dust cleared.
> This didn't seem to be the case throughout all Friday, so I didn't make a move
> yet.

Marco; can you post your two patches to bug#441974 and get them reviewed asap? The tree is going to be closed for another few days while we create the beta2 builds. However, if your changes are only touching tests, and are already r+'d, then I'll see if we might be allowed to land.
I'm trying, but the person who usually reviews such patches is involved in his ph.d process and is on and off rather randomly.
John, I got review. What's the next step in this special case?
(In reply to comment #69)
> John, I got review. What's the next step in this special case?

Marco: I see you've already landed bug#441974 on mozilla-central. Cool.

Next step is for Lukas to verify if your fix commented out the last of the failing tests, and if the a11y suite is now passing cleanly. If so, then we could finally be ready to enable a11y in production.
From today's dev meeting, now is a good time to enable a11y in production. However, this afternoon, we decided to not enable today because of the unrelated ESX server woes causing fun on the tree. Instead, planning to enable a11y test suite early tomorrow morning. 

ToDo:
1) joduinn to post patch uncommenting existing a11y step in moz2-unittest/master.cfg. This will be for win2 and linux, on both mozilla-1.9.1 and mozilla-central.
2) joduinn to reconfig master.cfg and watch to make sure it goes smmothly.
3) Marco to watch tests in a11y suite be run per build. In case any specific a11y test fails, Marco will land patch to disable that specific test, and work with developers to correct.  

Marco/Lukas/anyone: anything else we need to do?
Attachment #352249 - Flags: review?(ccooper) → review+
Comment on attachment 352249 [details] [diff] [review]
re-enable a11y for linux, win32 on both mozilla-central and mozilla-1.9.1

Requesting that this be backed out and the a11y tests disabled again. We are getting a lot of strange failures on both platforms that is impossible to deal with in the short term.
OK, I have reverted the local change (!!) which applied attachment 352249 [details] [diff] [review], and reconfig'd moz2-master.
(In reply to comment #74)
> OK, I have reverted the local change (!!) which applied attachment 352249 [details] [diff] [review], and
> reconfig'd moz2-master.

Thanks Nick.

I've reopened bug#441974, to figure out why some tests within the a11y suite started failing again.
We're going to turn these on in production tomorrow - stay tuned.
a11y test suite is now running in production on both 1.9.1 and m-c with success all around.

Closing.
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
Moving closed Future bugs into Release Engineering in preparation for removing the Future component.
Component: Release Engineering: Future → Release Engineering
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: