970886 - MarionetteTestRunner runs tests twice if the filename is the same

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect, P2)

Description

[Zac C (:zac)]

Here we have two py files with identical names.
Their class names and test methods are different, however when running the second test file the TestRunner will also run the test case from the first file again.


$ gaiatest --address localhost:2828 --testvars testvars.json --html-output results/index.html tests/app1/ tests/app2/

starting httpd
running webserver on http://10.246.27.126:53424/
test_do_something_app1 (test_case_1.App1) ... ok

----------------------------------------------------------------------
Ran 1 test in 10.833s

OK
test_do_something_app1 (test_case_1.App1) ... ok
test_do_something_app2 (test_case_1.App2) ... ok

----------------------------------------------------------------------
Ran 2 tests in 22.684s

OK

SUMMARY
-------
passed: 3
failed: 0
todo: 0

Comment 1

[Zac C (:zac)]

Attachment: bug_970886.zip

Here is the folder and tests I used for this

Comment 2

[Zac C (:zac)]

To prove real-world scenario, this arose where we had:

accessibility/lockscreen/test_unlock_to_homescreen.py
functional/lockscreen/test_unlock_to_homescreen.py

Comment 3

[Malini Das [:mdas] - Away, not checking bugmail]

Ha!

Elevating to major, this should never happen!

Comment 4

[Julien Pagès (:parkouss)]

Attachment: 970886.patch

I tried locally with the example provided in the zip (just modified the import to use MarionetteTestCase instead of gaia*, but the issue was the same).

So without the patch, the first test is ran twice, and the log shows 3 executed tests.

With the patch, this is fixed and we execute only 2 tests, as expected.

Comment 5

[Julien Pagès (:parkouss)]

Pushed to try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea76a506aea7

Comment 6

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/13388633bfc4

Comment 7

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/13388633bfc4