Closed Bug 1859035 Opened 5 months ago Closed 4 months ago

Investigate if tests of .popup-notification-primary-button click in Add-ons Manager need to be excluded from a11y_checks

Categories

(Toolkit :: Add-ons Manager, task, P3)

task
Points:
3

Tracking

()

RESOLVED FIXED
121 Branch
Tracking Status
firefox121 --- fixed

People

(Reporter: ayeddi, Assigned: rpl)

References

(Blocks 1 open bug)

Details

(Keywords: access, Whiteboard: [addons-jira])

Attachments

(2 files)

There are 12 failing tests for the button.popup-notification-primary-button that appear to test the Add-ons installation popup that sometimes is triggering AccessibilityUtils test for the click event while an add-on is still being installed and the popup is showing a progress bar (refer to the auto screenshot of toolkit/mozapps/extensions/test/xpinstall/browser_containers.js failure attached) - on this stage the primary button is still disabled, thus Node is not focusable via the accessibility API error is recorded by a11y_checks.

On the markup side of the popup notification, the element flagged is a <button> and manual testing on Fx Nightly 120 confirms that, when an add-on installation is completed, this button is focusable with keyboard and can be activated with keyboard alone.

a11y_checks failing data (with stack trace):

Result	Path	Error	Element ID	Element tagName	Element className	Stack trace
FAIL	browser/base/content/test/webextensions/browser_permissions_installTrigger.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/browser/base/content/test/webextensions/head.js:runOnce:389
chrome://mochitests/content/browser/browser/base/content/test/webextensions/head.js:testInstallMethod:423
chrome://mochitests/content/browser/browser/base/content/test/webextensions/browser_permissions_installTrigger.js:null:29
chrome://mochikit/content/browser-test.js:handleTask:1134
chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1206
chrome://mochikit/content/browser-test.js:Tester_execTest:1348
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/browser/browser_html_list_view_recommendations.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/browser/browser_html_list_view_recommendations.js:testListRecommendations:158
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/browser/browser_html_list_view_recommendations.js:testThemeList:198
chrome://mochikit/content/browser-test.js:handleTask:1134
chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1206
chrome://mochikit/content/browser-test.js:Tester_execTest:1348
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
Extension loaded
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_amosigned_trigger_iframe.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_amosigned_trigger_iframe.js:test:13
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_auth4.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_auth4.js:test:18
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_bug540558.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_bug540558.js:test:9
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_containers.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_containers.js:test:45
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js:test_allowedInstallDomain:545
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js:null:1543
chrome://mochikit/content/browser-test.js:handleTask:1134
chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1206
chrome://mochikit/content/browser-test.js:Tester_execTest:1348
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:339
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:382
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:667
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:635
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleClick:751
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js:test_incognito_checkbox:1263
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_doorhanger_installs.js:null:1540
chrome://mochikit/content/browser-test.js:handleTask:1134
chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1206
chrome://mochikit/content/browser-test.js:Tester_execTest:1348
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_hash2.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_hash2.js:test:10
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_httphash4.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_httphash4.js:test:10
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_installchrome.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_installchrome.js:test:9
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_unsigned_trigger_iframe.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_unsigned_trigger_iframe.js:test:13
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"
FAIL	toolkit/mozapps/extensions/test/xpinstall/browser_unsigned_url.js	Node is not focusable via the accessibility API		button	popup-notification-primary-button primary panel-footer-button	"
chrome://mochikit/content/browser-test.js:test_ok:1583
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:a11yFail:342
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertFocusable:385
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:assertCanBeClicked:668
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:runIfA11YChecks/this.AccessibilityUtils:638
chrome://mochikit/content/tests/SimpleTest/AccessibilityUtils.js:handleEvent:741
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:popupReady:295
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:handleEvent:306
EventListener.handleEvent*chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/head.js:setup:143
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_unsigned_url.js:runTest:20
resource://testing-common/SpecialPowersChild.sys.mjs:pushPrefEnv:850
chrome://mochitests/content/browser/toolkit/mozapps/extensions/test/xpinstall/browser_unsigned_url.js:test:5
chrome://mochikit/content/browser-test.js:Tester_execTest:1350
chrome://mochikit/content/browser-test.js:nextTest/<:1123
chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
"

Total are 15 tests that are failing for these components

Points: --- → 3
Whiteboard: [addons-jira]

While investigating more the a11y intermittent failure being hit using exploratory try pushes (because the failure couldn't be hit locally), we determined
that when a failure was being hit, the panel state was still "showing" (instead of "open") by the time the AccessibilityUtils were about to check
if the clicked element (a panel button) was focusable.

After looking into a couple other approaches to make Harness.popupReady to wait for the panel to get to the "open" state (which were not working and ended up
triggering other issues, like test timeouts), I tried the approach proposed in this patch (after reading some more of the related internals and noticed that
we emit a popup-shown topic from nsXULPopupShownEvent::Run right after having set the panel state to "open" a few lines before, see
https://searchfox.org/mozilla-central/rev/ffdc4971dc18e1141cb2a90c2b0b776365650270/layout/xul/nsMenuPopupFrame.cpp#404,411).

The push to try using that approach did pass successfully (as in "it didn't hit the a11y related failures it was hitting before"):

And the temporary debugging logs added in that try push confirmed that we were hitting the panel in the "showing" state at least a few times:

Assignee: nobody → lgreco
Status: NEW → ASSIGNED
Pushed by luca.greco@alcacoop.it:
https://hg.mozilla.org/integration/autoland/rev/cda838e04db1
Prevent AOM tests to hit intermittent failures in a11y jobs due to Harness.popupReady clicking to panel buttons while panel state is still "showing". r=ayeddi,robwu
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → 121 Branch
See Also: → 1863815
See Also: → 1864240
You need to log in before you can comment on or make changes to this bug.