1129073 - firefox-ui-tests should accept files that don't start with "test_" as arguments at the command line

Status: RESOLVED WONTFIX

(Testing :: Firefox UI Tests, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

This is a pretty confusing paper cut (:whimboo and myself hit this last quarter). The marionette test case class defines a regex that a file path must match to be considered a test it will actually run: https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/marionette_test.py#601

Without a match, the test isn't run, and no error is provided, even if the file is the only argument at the command line! We're manifest based in firefox-ui-tests, we should allow any sort of file name.

Comment 1

[Chris Manchester (limited bugmail, email directly)]

Attachment: Don't impose the match_re from MarionetteTestCase on FirefoxTestCase

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8558688 [details] [review]
Don't impose the match_re from MarionetteTestCase on FirefoxTestCase

I would like to stay as close as possible to the default behavior of Marionette, which is also what all the other test suites use at Mozilla. We shouldn't make an exception, which would only confuse people.

As talked with Chris on IRC we can let Marionette spit out an error, in case of a test file does not start with `test_`.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I filed bug 1129122 to get it fixed in Marionette proper.