1665751 - not running textselection/* results in failing tests

Status: NEW

(Core :: Disability Access APIs, defect, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

here is a try push:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=MfpS9pQtQNOP-UqJT63BtQ.0&revision=40f8e9e2f1df6e46bd55bfc831ba59649a3e2f39

and these two failures occur when I skip the 2 tests in testselection:
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/tree/test_img.html | Test timed out.
TEST-UNEXPECTED-FAIL | accessible/tests/mochitest/treeupdate/test_imagemap.html | Test timed out.

there are two tests:
[test_general.html]
[test_userinput.html]

If I only run test_general.html, I get a failure.
If I only run test_userinput.html, I get no failures.

this indicates that test_userinput.html is causing the browser to be in a state which future tests depends on, or that test_general.html puts the browser in a state that test_userinput.html resets.

https://searchfox.org/mozilla-central/source/accessible/tests/mochitest/textselection/test_userinput.html

I was not able to see any issues running these tests locally on my windows 10 laptop, but on try this fails on all platforms.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I would recommend disabling one or both of these tests so we don't end up with odd test dependencies in tree and can schedule appropriate tests for a11y (we are trying to switch to manifest based scheduling which could exclude some tests based on historical data of failures and code changes)

Comment 2

[James Teh [:Jamie]]

We saw these tests fail on autoland; see bug 1520779 comment 16. Does that mean autoland doesn't run these text selection tests sometimes or does that suggest that this isn't related to the selection tests?

Comment 3

[James Teh [:Jamie]]

It seems that creation of image map accessibles is unreliable; see bug 799781. In those tests, we move the mouse over the image map to force its creation (bug 745788), which has been sufficient until now. Perhaps something changed recently which makes even this hack unreliable?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

for context, we had switched to manifest based scheduling instead of task based scheduling. At the manifest level we select the most relevant manifests to run, not the most relevant tasks to run. A task would typically run 10-40 manifests, where at the manifest level we usually run 25-50% of the manifests in a given task.

In this specific case that we saw on autoland, we had removed 6 manifests from a11y and ran into this problem, that was narrowed down to removing test_userinput.html and causing the same problem.

I don't mind pushing a few try pushes, how far back in history do we want to go to test for recent breakage?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I went back a few weeks to:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=decision&revision=61ed3192760a3aaef282c089c064fc8dd3890125

and it still fails with the same errors

Comment 6

[Marco Zehe (:MarcoZ)]

Joel, I am shortly going to upload a patch for that file which converts it from our old event queue based approach to async/await. Can you do a try push to see if that solves the problem with the manifests?

Comment 7

[Marco Zehe (:MarcoZ)]

Attachment: Bug 1665751 - Convert test_userinput.html to promisified events, r?yzen!

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

yes, I have a push and should have results in ~30 minutes:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4eaa865114cfd75bbd1d7dc98d6ea4feadf0cb5f

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

unfortunately this change didn't fix the problem

Comment 10

[Marco Zehe (:MarcoZ)]

Even though this particular doesn't fix the bug yet, it is still good to have it because it gets rid of old cruft.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6917838bb773
Convert test_userinput.html to promisified events, r=yzen DONTBUILD

Comment 12

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/6917838bb773

Comment 13

[Marco Zehe (:MarcoZ)]

Attachment: Bug 1665751 Part 2: Factor out the Select All And Focus element sequence into a utility function for use by other test files, r?yzen!

Comment 14

[Marco Zehe (:MarcoZ)]

Attachment: Bug 1665751 Part 3: Rewrite textSelection/test_general.html to use async await instead of event queue, r?yzen!

Depends on D90859

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac97b9243102
Part 2: Factor out the Select All And Focus element sequence into a utility function for use by other test files, r=yzen DONTBUILD
https://hg.mozilla.org/integration/autoland/rev/9d74ce5f7e04
Part 3: Rewrite textSelection/test_general.html to use async await instead of event queue, r=yzen DONTBUILD

Comment 16

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/ac97b9243102
https://hg.mozilla.org/mozilla-central/rev/9d74ce5f7e04

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:Jamie, maybe it's time to close this bug?

Comment 18

[James Teh [:Jamie]]

The patches landed here clean things up (which is still useful), but unfortunately didn't fix the failure. So, this bug is still valid and should remain open.

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.