Closed Bug 441974 Opened 16 years ago Closed 16 years ago

Several tests fail when running in conjunction with other mochitests, succeed when running stand-alone

Categories

(Core :: Disability Access APIs, defect)

Product:
Core
Component:
Disability Access APIs
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla1.9.1

People

(Reporter: MarcoZ, Assigned: MarcoZ)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Keywords: access, verified1.9.1)

Attachments

(3 files, 1 obsolete file)

Disable failing tests and provide bug numbers.
3.48 KB, patch
surkov
: review+
Details | Diff | Splinter Review
Fix last remaining mochitest failures
3.65 KB, patch
surkov
: review+
beltzner
: approval1.9.1+
Details | Diff | Splinter Review
one more fix
3.57 KB, patch
MarcoZ
: review+
Details | Diff | Splinter Review 
This is a spin-off of bug 424195. Several Mochitests that succeed when running stand-alone.

1. Make a release or debug build with the enable_tests option.
2. Launch runtests.py with the --a11y switch.
3. In the test harness screen, click the test_bug368835.xul file.

Result: All tests pass.

4. Return to the main harness screen.
5. Click "Run all tests".
6. Observe as some of the tests for test_bug368835.xul fail.

Same result for (maybe):
test_bug420863.html
test_nsOuterDocAccessible.html
test_nsIAccessibleText.html

Notice that clicking the "Run tests" several times and watching the results, they appear randomly, except for the test_table_indexes.html, which will be dealt with in bug 437980, and the test_nsIAccessibleImage.html, which gets fixed in bug 439057. All others ordinarily pass or have todo items, but when running all together, the above candidates tend to fail.

Any idea why?
On IRC on the #developers today, Shawn recommended that our seemingly random failures could have something to do with the tests requiring a certain state when they start running.

The one thing that is identical in all tests is this: In order to be able to retrieve accessibles for any given DOM element, we have to enable the nsIAccessibleRetrieval service:
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_aria_activedescendant.html?force=1#14,
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_aria_activedescendant.html?force=1#17,
and
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_aria_activedescendant.html?force=1#34.

So, whenever a new test file starts, we do a call to this getService method.

If there were a way to discover whether the nsIAccessibleRetrieval service is already active, I'd be happy to use that method to avoid unnecessary calls.

Any thoughts?
That's likely not your problem.
Just talked to Ted M. on IRC, and he suggested to take a look at this under Valgrind (Windows) or Purify (Linux).

Anyone familiar with these tools?
sayrer, any chance you could take a stab at this? Ted M. suggested you might be able to find something.
Clint, Sayrer, any chance you'd be able to take a stab at this this week? Would be very much appreciated!
Depends on: 445454
Could this have something to do with the test being run inside an iframe vs. as the toplevel document?
Hi Ted, this is the bug we were discussing yesterday about the a11y random failures.
This fixes the tests all except for test_textattrs.html, which has 6 failures on the input type text having to do with different default fonts.

Lukas, do these tests also work on the staging if you apply this patch?
Assignee: nobody → marco.zehe
Status: NEW → ASSIGNED
Attachment #334860 - Flags: review?(surkov.alexander)
Comment on attachment 334860 [details] [diff] [review]
Fix some of the tests to pass when running in conjunction with others

><html><body><pre style="word-wrap: break-word; white-space: pre-wrap;">diff --git a/accessible/tests/mochitest/test_bug368835.xul b/accessible/tests/mochitest/test_bug368835.xul
>--- a/accessible/tests/mochitest/test_bug368835.xul
>+++ b/accessible/tests/mochitest/test_bug368835.xul
>@@ -100,22 +100,22 @@
>       gTreeInvalidatedCount++;
> 
>       switch (gTreeInvalidatedCount) {
>         case 1:
>           TreeInvalidatedHandlerHelper(aEvent, 0, 5, null, null,
>                                        "nsITreeBoxObject::rowCountChanged");

iirc TreeInvalidatedHandlerHelper takes null because aEvent.getData() should return null if there is no requested data. So does it return empty string in this case?

>-      testThis("emptyLink", 93, 4, "");
>+      testThis("emptyLink", 93, 4, null);
> 
>           // see bug 428954: No name wanted for internal frame
>-          is(outerDocAcc.name, "", "Wrong name for internal frame!");
>+          is(outerDocAcc.name, null, "Wrong name for internal frame!");
> 

because it's a bit tricky where we use empty string or null then please comment here why do we return null

would be nice to look one more patch :)
Attachment #334860 - Flags: review?(surkov.alexander)
I simply took my output from the logs the tests produced and transformed them into expected values. So, the event.getData calls really return 0 instead of null, and in the other cases return null instead of "". Bug 451533 was actually spun off of this work because I saw that we're not always truncating the aName parameter, but this doesn't help those tests. Some of them still return null instead of "".
Lukas, can you give us the lines that show failures from a the staging machine on Linux? I'd like to see if we're getting the same or different failures there. Thanks!
Marco: fyi, Lukas is on vacation and offline until approx 08 sept. 

Robcee or myself can try to help, but we dont have context. Can you clarify which lines of which logs are you looking for?
(In reply to comment #12)
> Robcee or myself can try to help, but we dont have context. Can you clarify
> which lines of which logs are you looking for?

Basically a current run of the whole a11y test suite, all lines with "unexpected" in them, from the Linux staging box.
Depends on: 454636
Depends on: 454647

        Attachment #334860 -
        Attachment is obsolete: true

        Attachment #337930 -
        Flags: review?(surkov.alexander)

        Attachment #337930 -
        Flags: review?(surkov.alexander) → review+

    
    

    
  
Pushed in changeset:
http://hg.mozilla.org/mozilla-central/rev/ffc759affe23
Status: ASSIGNED → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED

    
    

    
  
With this checkin, I'm still getting two test failures in my build (from just a little bit ago) on Windows:
32 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_b
ug420863.html | Error thrown during test: uncaught exception: [Exception... "Com
ponent returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numAc
tions]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrom
e://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 :: line 97"
 data: no] - got 0, expected 1
833 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_
nsIAccessible_focus.html | No focused state for element with ID aria-link.

    
    

    
  
Reopening, as there are still a couple of failing tests reported in comment#16.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
So I tried turning on the a11y tests in moz2 unittests.

Here is what the results were:

Win32 (7 tests fail consistently):

*** 851 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No
focused state for element with ID aria-link2.
*** 919 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area1 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 935 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area2 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 980 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area4 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 998 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area5 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 1017 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area6 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
*** 1048 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area7 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"

On Linux (22 fails consistent):

79:*** 32 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_bug420863.html | Error thrown
during test: uncaught exception: [Exception... "Component returned failure
code: 0x80004005 (NS_ERROR_FAILURE) [nsIAccessible.numActions]"  nsresult:
"0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame ::
chrome://mochikit/content/a11y/accessible/test_bug420863.html :: doTest3 ::
line 97"  data: no] - got 0, expected 1
153:*** 112 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item1.1 - got "", expected "1"
154:*** 113 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item1.1 - got "", expected "1"
156:*** 115 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item1.2 - got "", expected "1"
157:*** 116 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item1.2 - got "", expected "3"
159:*** 118 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item1.4 - got "", expected "2"
160:*** 119 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item1.4 - got "", expected "3"
162:*** 121 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong posinset
on menu_item2 - got "", expected "3"
163:*** 122 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_groupattrs.xul | Wrong setsize
on menu_item2 - got "", expected "3"
419:*** 385 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImage! - got 587, expected 588
441:*** 407 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageWithAlt! - got 713, expected 714
463:*** 429 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageWithTitle! - got 839, expected 840
485:*** 451 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageEmptyAlt! - got 965, expected 966
507:*** 473 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleImage.html | Wrong
screen y coordinate for linkedImageEmptyAltAndTitle! - got 1091, expected 1092
563:*** 530 ERROR TEST-UNEXPECTED-PASS |
chrome://mochikit/content/a11y/accessible/test_nsIAccessibleTable_1.html |
columnHeader should not throw
876:*** 850 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_nsIAccessible_focus.html | No
focused state for element with ID aria-link.
942:*** 920 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area1 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
958:*** 936 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area2 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1003:*** 981 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area4 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1021:*** 999 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area5 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1040:*** 1018 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area6 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"
1071:*** 1049 ERROR TEST-UNEXPECTED-FAIL |
chrome://mochikit/content/a11y/accessible/test_textattrs.html | Attribute
'background-color 'has wrong value. Getting default attributes for area7 - got
"rgb(0, 0, 0)", expected "rgb(255, 255, 255)"

    
  
Depends on: 455834

    
  
Depends on: 455840

    
    

    
  
OK, I believe I fixed the ones Lukas is seeing on Windows by disabling, temporarily fixing with hard-coded values, or marking as known fail. See dependent bugs for more details on these.

I remember having seen that "uncaught exception" error once, and Ted seems to see it, too, but I can't reproduce that right now any longer. The root cause of that seems to be that removing the event listener for the "click" event somehow invalidates the accessible.

Anyone got an idea why the offsets for those failing y coordinates on Linux are all off by 1? Are the offsets on Linux 1-based, not 0-based or something?

    
    

    
  
Marco, any chance we can just disable / exclude the failing tests in comment#18, while debugging continues? We've a whole bunch of unittest systems migration work on hold because of this. :-(

    
  
Depends on: 456344

    
  
Depends on: 456346

    
  
Depends on: 456347

    
    

    
  
I've wallpaper-fixed the test failures Lukas provided from the Windows and Linux logs by mostly disabling/protecting against them and providing bug numbers for these specific failures.

Lukas or John, could you run them again on the staging machines and see if you get any additional failures?

    
    

    
  
Marco - I am waiting on some VMs to put on m-c staging (see bug 455433 and 455431) at which point I will be able to run these on m-c again.

Is it possible for you to look at the tests that are failing in 1.9 at this time and disable any tests there that are failing?

    
    

    
  
Lukas, could you provide me with a full failure log when you run the a11y tests on 1.9 staging on both Windows and Linux? I don't trust my failures I'm getting locally, so would like to use your output to wall-paper-fix/disable them.

    
    

    
  


  
This patch does two things:
1. It corrects test cases for select and select@sioze elements which have different names after bug 455482 was checked in. These tests should have been fixed as part of that bug, but weren't.
2. Disable checking for attributeChanged events on anything but Windows. Linux fails with this test.

        Attachment #349938 -
        Flags: review?(surkov.alexander)

    
    

    
  
Comment on attachment 349938 [details] [diff] [review]
Fix last remaining mochitest failures


>-          is(gA11yEventObserver.mTextAttrChangedEventCounter, 2,
>-             "Wrong count of 'text attribute changed' events for " + ID);
>+          if (navigator.platform == "Win32")
>+            is(gA11yEventObserver.mTextAttrChangedEventCounter, 2,
>+               "Wrong count of 'text attribute changed' events for " + ID);
>+          else
>+            todo(gA11yEventObserver.mTextAttrChangedEventCounter, 2,
>+                 "Wrong count of 'text attribute changed' events for " + ID);

If it's fail on linux and mac both then please file bug for this (if we don't have it already).

        Attachment #349938 -
        Flags: review?(surkov.alexander) → review+

    
    

    
  
Done, bug 466629.

    
    

    
  
Comment on attachment 349938 [details] [diff] [review]
Fix last remaining mochitest failures

a191=beltzner

        Attachment #349938 -
        Flags: approval1.9.1? → approval1.9.1+

    
    

    
  
http://hg.mozilla.org/mozilla-central/rev/136ac0be7eda
Status: REOPENED → RESOLVED
Closed: 16 years ago → 16 years ago
Resolution: --- → FIXED

    
    

    
  
We tried running a11y in production this morning, and had a bunch of a11y tests fail again. Unclear why some tests are failing like this, and MarcoZ/joduinn unable to figure it out, so we backed a11y out from production.

Reopening. :-(
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

    
    

    
  
I had a very quick look at one of the failing tests:
http://mxr.mozilla.org/mozilla-central/source/accessible/tests/mochitest/test_events_caretmove.html?force=1

I notice the use of "this" to declare variables in the function definitions. Variables declared via "this.foo" inside a function will not be function local. In these cases "this" is most likely the window object (global namespace). I wasn't sure if this was known/intended? If you want function local scope use "var foo"...

I'm not familiar with the mochitest framework... but I'm wondering if there is something funny going on with the global namespace getting cluttered?

    
    

    
  
Surkov, of course you are correct, thanks! I see they are indeed been used as object constructors. Please disregard comment #30. I'm used to a different test harness.

    
    

    
  

      
      
      
      

        Attached patch
          
          
        one more fix
        — 
          — Splinter Review
      

    


  
this should help

        Attachment #352504 -
        Flags: review?(marco.zehe)

    
    

    
  
John, can we get this patch onto both a Windows and Linux staging server and run a build cycle with a11y enabled there to see if this improves the situation? Since I cannot reproduce any of this locally, we'd have to actually run it in that environment to get accurate results.

    
    

    
  
CC'ing Bob to see if he has any expertise to share!

    
    

    
  
Run on m-c staging with the patch applied locally, a11y on win32 isn't crashing out anymore. Both Linux and Windows just have the one tests that is failing/passing unexpectedly:

Win32

*** 1007 ERROR TEST-UNEXPECTED-FAIL | chrome://mochikit/content/a11y/accessible/test_textattrs.html | Wrong count of 'text attribute changed' events for area8 - got 0, expected 2

Linux

*** 1007 ERROR TEST-UNEXPECTED-PASS | chrome://mochikit/content/a11y/accessible/test_textattrs.html | 2 - Wrong count of 'text attribute changed' events for area8

    
    

    
  
the patch i'm referring to is the "one more fix" one in the attachments for this bug.

    
    

    
  
I stand corrected, the windows build just passed a11y tests with this patch applied locally.  So it's just the unexpected pass on linux.  Marco - if you want to approve this and get it checked in, I can enable a11y tests tomorrow.

    
    

    
  
Ok, now the linux build just passed a11y tests and the Windows build failed that single test same as in comment 36.

Not sure what's happening here.

    
    

    
  
I've set them to output to http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest

You want to look at Linux moz2-linux-slave09 dep unit test  and WINNT 5.2 bm-win2k3-unittest-01 dep unit test

    
  

        Attachment #352504 -
        Flags: review?(marco.zehe) → review+

    
    

    
  
Comment on attachment 352504 [details] [diff] [review]
one more fix

Thanks Lukas! I'm sure the test fluke with that particular test can be easily resolved.

Alex, please take a look at that particular test and see if you can correct it. Check it in as a bustage fix.

    
    

    
  
(In reply to comment #36)
> Run on m-c staging with the patch applied locally, a11y on win32 isn't crashing
> out anymore. Both Linux and Windows just have the one tests that is
> failing/passing unexpectedly:
> 
> Win32
> 
> *** 1007 ERROR TEST-UNEXPECTED-FAIL |
> chrome://mochikit/content/a11y/accessible/test_textattrs.html | Wrong count of
> 'text attribute changed' events for area8 - got 0, expected 2
> 
> Linux
> 
> *** 1007 ERROR TEST-UNEXPECTED-PASS |
> chrome://mochikit/content/a11y/accessible/test_textattrs.html | 2 - Wrong count
> of 'text attribute changed' events for area8

I guess it's bug 466629

    
    

    
  
This patch landed along with the fix for bug 448744. In addition, bug 466629 landed, which fixes Lukas' above reported random orange. I believe we're good! I don't see any failures when I run these locally.
Status: REOPENED → RESOLVED
Closed: 16 years ago → 16 years ago
Resolution: --- → FIXED

    
    

    
  
Yes, we are definitely seeing green in both m-c and 1.9.1 Linux staging, and in m-c Win32.  I have to see a build of 1.9.1 Win32 just to verify - but this looks really promising.

    
    

    
  
Lukas, we didn't push anything to m-1.9.1 yet, this was all exclusively m-c work. So, m-1.9.1 is still subject to the same randomness that we've been seeing all along.

    
    

    
  
Alexander, logs on the mozilla-1.9.1 staging servers show similar failures as the "One more fix" patch fixes. However this patch does not apply at all to the 1.9.1 branch, since there were other changes to common.js which cause it not to apply.

Can you create a patch for mozilla-1.9.1 that fixes the problems?

    
    

    
  
I created a smaller patch that just contains the changes to test_events_caretmove.html and pushed that to mozilla-1.9.1. The rest of that patch applies to features  that were explicitly added to mozilla-central. We'll see if this was enough to fix the Windows timeouts we're seeing on mozilla-1.9.1.
Keywords: fixed1.9.1

    
    

    
  
MarcoZ, would be nice to have patch attached or to have changeset at least.

    
    

    
  
(In reply to comment #48)
> MarcoZ, would be nice to have patch attached or to have changeset at least.

Oops, sorry! Here's the changeset: http://hg.mozilla.org/releases/mozilla-1.9.1/rev/37d44cd6a2c5

    
    

    
  
Tests were enabled on December 18, 2008, and have been running continuously since.
Keywords: fixed1.9.1 → verified1.9.1

    
    

    
  
Same goes for mozilla-central AKA trunk.
Status: RESOLVED → VERIFIED

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: