1836727 - "mach test" fails for Marionette tests outside the unit test folder

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect, P3)

Description

[:Gijs (he/him)]

 % ./mach test browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py
/path/to/objdir/backend.TestManifestBackend is out of date with respect to /path/to/objdir/config.status
Test configuration changed. Regenerating backend.
UNKNOWN TEST: browser/components/sessionstore/test/marionette/test_restore_windows_after_restart_and_quit.py
I was unable to find tests from the given argument(s).

You should specify a test directory, filename, test suite name, or
abbreviation.

It's possible my little brain doesn't know about the type of test you are
trying to execute. If you suspect this, please request support by filing
a bug at
https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=General.

Running ./mach marionette-test with exactly the same path works fine. If it matters, this is an artifact build.

(unclear if this is mac-specific but that's where I'm seeing it at the moment)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Interesting that this is failing but this is probably outside of our scope for Marionette and a generic issue with the mach test command.

Andrew or Geoff, do you know where this issue could actually come from?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I assume that mach test only accepts Marionette tests from testing/marionette/harness/marionette_harness/tests/unit/?

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #2)

I assume that mach test only accepts Marionette tests from testing/marionette/harness/marionette_harness/tests/unit/?

Actually, I think some other marionette test directories are okay. I think this directory just needs to be linked in via moz.build.

Comment 4

[Geoff Brown [:gbrown]]

Attachment: Bug 1836727 - Link marionette test manifest in browser/components/sessionstore

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that the revision from Geoff has been abandoned here the reply from Gijs so we don't miss the details:

In D180265#5951020, @whimboo wrote:

Thanks for figuring this out Geoff! But this patch would only fix the case for sessionstore, and we have other tests as well on different location which don't work as well. I've only tested a few of them which are listed at:
https://searchfox.org/mozilla-central/source/testing/marionette/harness/marionette_harness/tests/unit-tests.ini

So it would require more changes to moz.build files for all the other components as well.

I'm not really familiar with this, but is this suggesting that for any new manifest we'd need to both add a moz.build entry and update the linked unit-tests.ini to reference the same file? Because that seems kind of tedious...

The way other tests work is (I believe) that there is only a moz.build entry, and there isn't a single central ini file that lists all the other ini files. Could marionette use a similar system? ISTM that might potentially make e.g. chunking or otherwise treating these tests similar to mochitests/xpcshell/... easier as well...

If switching to the moz.build registration for ini files is not possible for some reason, perhaps we can teach the mach test bits to learn to read the include format for the marionette stuff, instead of having to duplicate the registrations? But that does seem very marionette-specific...

Given that this is specific to Marionette and how we collect the tests we should actually handle it in the Marionette client and harness component.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We probably need similar patches as https://phabricator.services.mozilla.com/D196188 to get this issue fixed for all the known Marionette test locations: https://searchfox.org/mozilla-central/source/testing/marionette/harness/marionette_harness/tests/unit-tests.ini

Maybe I have time by the end of this months to do this work.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1836727 - [marionette] Fix MARIONETTE_MANIFESTS usage in moz.build files.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0516138a30ac
[marionette] Fix MARIONETTE_MANIFESTS usage in moz.build files. r=webdriver-reviewers,necko-reviewers,application-update-reviewers,search-reviewers,sessionstore-reviewers,places-reviewers,dom-storage-reviewers,Standard8,jesup,jdescottes,nalexander,sclements,janv

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/0516138a30ac