Closed Bug 777935 Opened 9 years ago Closed 9 years ago

Marionette tests can be flaky because previous tests dont clean up

Categories

(Testing :: Marionette, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla17

People

(Reporter: automatedtester, Assigned: automatedtester)

References

Details

Attachments

(1 file)

I have noticed that when running the entire test suite we are getting some flakiness. 

E.g. Window Management test counts the amount of open windows when asserting and had 1 more than expected which appears to be from the test that ran before

We should probably restart the browser between test modules
Is there actually 1 extra window open, or are we holding a stale reference?
Yes, I see the same problem; basically Marionette gets confused when switching windows sometimes, and switches to the wrong one.  It may be a timing issue.  Definitely needs to be fixed.
Blocks: 770769
On my machine, this problem is easily reproduced if Firefox is in the foreground while the tests run.  If it's in the background, obscured by a Terminal window for example, it does not reproduce.

I'm guessing this is a timing issue.
So it looks like this is what's happening:

- some tests use window.open in an execute_script to open a new window to run a test, e.g., http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_findelement.py#131
- the test then continues immediately, without waiting for the window to load
- sometimes, this means that the test will be executed before the window is really open
- then, when the test teardown happens (e.g., http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_findelement.py#134), it will close the wrong window
Assignee: nobody → dburns
Attached patch add test cleanupSplinter Review
I tracked it down to a test file that opens a window and doesnt clean it up. I have added a tearDown to close all windows but the main one.
Attachment #646576 - Flags: review?(jgriffin)
Comment on attachment 646576 [details] [diff] [review]
add test cleanup

Review of attachment 646576 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!
Attachment #646576 - Flags: review?(jgriffin) → review+
https://hg.mozilla.org/mozilla-central/rev/52f5b60aa1d6
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla17
You need to log in before you can comment on or make changes to this bug.