1986422 - Manifest file ignored with a folder of tests specified as path to tests

Status: RESOLVED WONTFIX

(Testing :: Marionette Client and Harness, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

If you run Marionette tests by specifying a path like browser/components/sessionstore/test/marionette the manifest file as contained in this folder is not obeyed but all the tests found are getting run. This is problematic especially for try builds when you have to pass in a path to a folder and not a file like:

https://treeherder.mozilla.org/jobs?repo=try&revision=120438321620088501cdf673ef7ca8188adb6f3f

In this case the test browser/components/sessionstore/test/marionette/test_restore_windows_after_windows_shutdown.py is getting run not only on Windows as specified in the manifest file but on all platforms.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Instead of ignoring a manifest file we probably should take it into account by default, apply the expectations and settings and only run those tests as allowed. If users want to run all the tests we could add a --ignore-manifest argument.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1979374

:ahal, since you are the author of the regressor, bug 1979374, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I actually noticed the following description of the tests argument for Marionette:

https://searchfox.org/firefox-main/rev/3c23ce1368431d49bae08e8e211f7f2bf4e4829d/testing/marionette/harness/marionette_harness/runner/base.py#275-276

            "When a directory is specified, "
            "all test files in the directory will be run.",

So this was never a regression but an expected behavior. So we can revert the latest changes to this bug.

Nevertheless I actually wonder if we should still check for a manifest and set expectations even though it would only happen when an extra argument like --force-manifest is specified. We should get this discussed.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

When we do this we should pick up all the manifest files from the folder. But to be consistent with other harnesses we probably should keep what we have right now. Marking as wontfix.