1700112 - [meta] Fix tests that fail when browser.proton.enabled is true

Status: RESOLVED FIXED

(Firefox :: Theme, task, P1)

Description

[Mike Conley (:mconley) (:⚙️) (PTO Dec 22 - Jan 2)]

Gijs has a try push here: https://treeherder.mozilla.org/jobs?repo=try&revision=e8cca56e9dc93854d84d55bc91c04106a56be999

Comment 1

[:Gijs (out until Jan 5th; he/him)]

Fixing bug 1699310 might fix browser/base/content/test/performance/browser_startup_images.js , the pocket button work in bug 1693376 is likely to fix some/all of the CUI ones.

Comment 2

[:Gijs (out until Jan 5th; he/him)]

(In reply to :Gijs (he/him) from comment #1)

The pocket button work in bug 1693376 is likely to fix some/all of the CUI ones.

Re-pushing now that that's landed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3ed214b2325b5bf83933eaae85ee85ffb66fc768

Comment 3

[:Gijs (out until Jan 5th; he/him)]

(In reply to :Gijs (he/him) from comment #2)

(In reply to :Gijs (he/him) from comment #1)

The pocket button work in bug 1693376 is likely to fix some/all of the CUI ones.

Re-pushing now that that's landed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3ed214b2325b5bf83933eaae85ee85ffb66fc768

OK, so the pocket button definitely helped. I ran some more tests. The a11y test results look concerning, and I'm not sure off-hand why they would fail. Jamie, I don't suppose you (or someone on your team) could spare some time to help? To reproduce, flipping browser.proton.enabled to true in firefox.js and running ./mach build faster should be sufficient. try has a lot of failures along the lines of:

TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_listcontrols.xhtml | Test timed out.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_menu.xhtml | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_menu.xhtml | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_name.html | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_name.html | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_removal.html | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_removal.html | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_selects.html | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_tree.xhtml | Unique type focus event was handled.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_tree.xhtml | Unique type focus event was handled.

Comment 4

[James Teh [:Jamie] (away returning 12 Jan)]

(In reply to :Gijs (he/him) from comment #3)

TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_listcontrols.xhtml | Test timed out.

This one happens because the test clicks near the top left of a menulist and expects it to open. That seems to work with Proton disabled, but I guess some sizing changed somewhere and now a top left click doesn't work. Assuming that's intentional, I think it's reasonable to change the test to click at the centre, but that will require a tweak to some supporting code, since it doesn't support centre clicks at present.

TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/events/test_focus_menu.xhtml | Unique type focus event was handled.
...

The good news is that the rest of these failures are just side effects of the timed out test not cleaning up properly. With the issue above fixed, they go away.

Comment 5

[Marco Bonardo [:mak]]

I tried to flip the pref today for a Try run while investigating urlbar graduation, and I still see failures when just enabling the main proton pref. This was an artifact and using firefox.js, I'm not sure if any of these would be avoided with a full build, but it sounds like some failures may be real https://treeherder.mozilla.org/jobs?repo=try&revision=ecda7f2841a27082ea1f586533f64558f1043311

Comment 6

[:Gijs (out until Jan 5th; he/him)]

(In reply to Marco Bonardo [:mak] from comment #5)

I tried to flip the pref today for a Try run while investigating urlbar graduation, and I still see failures

It's more like "again"... people keep landing things (how dare they try to fix more bugs! :-) ) and we don't notice when they break tests because it's all preffed off...

Comment 7

[Mike Conley (:mconley) (:⚙️) (PTO Dec 22 - Jan 2)]

Proton is enabled by default in Nightly. We've got some intermittents, and tests where we still intentionally turn Proton off, but we can address those in bug 1698882.