777935 - Marionette tests can be flaky because previous tests dont clean up

Status: RESOLVED FIXED

(Testing :: Marionette, defect)

Description

[David Burns :automatedtester]

I have noticed that when running the entire test suite we are getting some flakiness. 

E.g. Window Management test counts the amount of open windows when asserting and had 1 more than expected which appears to be from the test that ran before

We should probably restart the browser between test modules

Comment 1

[Jonathan Griffin (:jgriffin)]

Is there actually 1 extra window open, or are we holding a stale reference?

Comment 2

[Jonathan Griffin (:jgriffin)]

Yes, I see the same problem; basically Marionette gets confused when switching windows sometimes, and switches to the wrong one.  It may be a timing issue.  Definitely needs to be fixed.

Comment 3

[Jonathan Griffin (:jgriffin)]

On my machine, this problem is easily reproduced if Firefox is in the foreground while the tests run.  If it's in the background, obscured by a Terminal window for example, it does not reproduce.

I'm guessing this is a timing issue.

Comment 4

[Jonathan Griffin (:jgriffin)]

So it looks like this is what's happening:

- some tests use window.open in an execute_script to open a new window to run a test, e.g., http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_findelement.py#131
- the test then continues immediately, without waiting for the window to load
- sometimes, this means that the test will be executed before the window is really open
- then, when the test teardown happens (e.g., http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_findelement.py#134), it will close the wrong window

Comment 5

[David Burns :automatedtester]

Attachment: add test cleanup

I tracked it down to a test file that opens a window and doesnt clean it up. I have added a tearDown to close all windows but the main one.

Comment 6

[Jonathan Griffin (:jgriffin)]

Comment on attachment 646576 [details] [diff] [review]
add test cleanup

Review of attachment 646576 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 7

[David Burns :automatedtester]

landed in http://hg.mozilla.org/integration/mozilla-inbound/rev/52f5b60aa1d6

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/52f5b60aa1d6