Closed Bug 1480142 Opened 2 years ago Closed 1 year ago

Support for running specific marionette test in isolation

Categories

(Firefox Build System :: Try, enhancement)

enhancement
Not set

Tracking

(firefox65 fixed)

RESOLVED FIXED
mozilla65
Tracking Status
firefox65 --- fixed

People

(Reporter: gabriel-v, Assigned: marco)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

I'm trying to run this command:

  ./mach try fuzzy toolkit/components/microformats/test/marionette/test_standards.py

And getting the response: 

  UNKNOWN TEST: toolkit/components/microformats/test/marionette/test_standards.py

Is running marionette tests like this supported?
If not, what are the steps to support this?
Flags: needinfo?(ahal)
Duplicate of this bug: 1480141
This is not a Marionette harness question. So moving into build system. Please adjust if the component is not the correct one.
Component: Marionette → Try
Product: Testing → Firefox Build System
The fact that you're getting UNKNOWN_TEST suggests that the build system doesn't know about it, which is strange. Basically this function isn't returning any results:
https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py#323

It's possible you need to have a build for it to get recognized.

However, even if we solved this, it looks like marionette isn't hooked up to |mach try fuzzy <path>| yet. This second part is easy to fix though and involves adding a task regex here:
https://searchfox.org/mozilla-central/source/testing/mozbase/moztest/moztest/resolve.py#68
Flags: needinfo?(ahal)
The tests under testing/marionette/harness/marionette_harness/ don't raise UNKNOWN TEST, just the mentioned above. I'll add the task regex for this bug, so it integrates nicely with bug 1429463.

When I'll do comprehensive testing for bug 1429463 I'll open bug reports for outliers like the one above.
Assignee: nobody → tvijiala
Status: NEW → ASSIGNED
After adding the task reged, selecting tests and directories with `mach try fuzzy` works now.

Failing try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1cf3738e84bc9727fa446c38dd5d714dbe2e283c

On linux,, macos and windows:
[task 2018-08-21T13:55:00.904Z] 13:55:00     INFO -  usage: runtests.py [options] test_file_or_dir <test_file_or_dir> ...
[task 2018-08-21T13:55:00.906Z] 13:55:00     INFO -  runtests.py: error: Test file(s) not found: browser/components/migration/tests/marionette

The path doesn't exist on the test machine probably because there's no complete source checkout. This could be fixed by rewriting the paths in MOZHARNESS_TEST_PATH to point to abs_marionette_tests_dir, here: https://dxr.mozilla.org/mozilla-central/rev/4e56a2f51ad739ca52046723448f3129a58f1666/testing/mozharness/scripts/marionette.py#310. I'd do an interactive task to see how the paths should be mapped, but I don't have the scopes to do that.

For Android, it's a different story:
[task 2018-08-21T13:58:56.684Z] 13:58:56     INFO -  usage: runtests.py [options] test_file_or_dir <test_file_or_dir> ...
[task 2018-08-21T13:58:56.685Z] 13:58:56     INFO -  runtests.py: error: unrecognized arguments: browser/components/migration/tests/marionette

It seems that the Android command parser does not support taking a directory with the tests. I don't get it why this is happening, since it's using the same argument parser (which supports the 'tests' argument). Maybe it's because of the argument ordering?
Flags: needinfo?(ahal)
Blocks: 1413928
(In reply to Tudor-Gabriel Vijiala [:tvijiala] from comment #5)
> I don't have the scopes to do that.

Actually I do, I was just logged out. Trying it out now.
I don't think the first error is happening due to lack of source checkout (none of the "test" tasks have a source checkout). Rather it's likely a bug in testing/mozharness/scripts/marionette.py around how the MOZHARNESS_TEST_PATHS env is handled.

If you download a "common.tests.zip" build artifact, you'll see all the marionette test files, e.g "marionette/tests/browser/components/migration/tests/marionette"

So possibly we're not mapping that relative path to the proper location in the extracted tests.zip. I'm not totally surprised this isn't working as this code path has likely never been hit until now. I haven't looked into the Android issue yet, but let's see if we can get it working on desktop first.
Flags: needinfo?(ahal)
Blocks: 1507108
No longer blocks: 1429463
Assignee: gabriel.vijiala → mcastelluccio
Attachment #9026010 - Flags: review?(ahal)
The paths that we were passing to the Marionette test runner were wrong, as it expects paths relative to abs_marionette_tests_dir.
Attachment #9026011 - Flags: review?(hskupin)
(In reply to Marco Castelluccio [:marco] from comment #9)
> Created attachment 9026011 [details] [diff] [review]
> Pass test paths relative to abs_marionette_tests_dir to the Marionette test
> runner
> 
> The paths that we were passing to the Marionette test runner were wrong, as
> it expects paths relative to abs_marionette_tests_dir.

Actually, relative to dirs['abs_test_install_dir']/marionette/tests, I need to update the commit message.
Attachment #9026010 - Flags: review?(ahal) → review+
Keywords: leave-open
I've landed the first patch in bug 1480141.
Depends on: 1480141
Keywords: leave-open
Comment on attachment 9026011 [details] [diff] [review]
Pass test paths relative to abs_marionette_tests_dir to the Marionette test runner

>+            paths = [os.path.join(dirs['abs_test_install_dir'], 'marionette', 'tests', p)

It would be great to get this into the `query_abs_dirs()` method, but that can be done at a later time if still necessary. 

Thanks for the fix!
Attachment #9026011 - Flags: review?(hskupin) → review+
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/eed5fefba089
Pass test paths relative to the Marionette test install dir to the Marionette test runner. r=whimboo
https://hg.mozilla.org/mozilla-central/rev/eed5fefba089
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
You need to log in before you can comment on or make changes to this bug.