1892038 - ./mach try cannot run specific marionette tests; marionette.py prepends build/tests/marionette/tests/ to the passed test paths

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect, P3)

Description

[Rob Wu [:robwu]]

When I use mach try with paths I get a bunch of failures in Mn and Mn-swr. It looks like Marionette tries to read the tests from the wrong location. All other test runners look up the test manifest at the given path, but Marionette prepends build/tests/marionette/tests/ before the paths:

Relevant code responsible:

• os.path.join(dirs["abs_test_install_dir"], "marionette", "tests", p)
  https://searchfox.org/mozilla-central/rev/b94e479d0b79b157029379832d05229df646e134/testing/mozharness/scripts/marionette.py#360-370
• dirs["abs_test_install_dir"] = os.path.join(abs_dirs["abs_work_dir"], "tests")
  https://searchfox.org/mozilla-central/rev/b94e479d0b79b157029379832d05229df646e134/testing/mozharness/scripts/marionette.py#222

Minimal example:

• ./mach try fuzzy -q "'64-qr" toolkit/components/extensions/test/marionette
• Generated try task config: https://hg.mozilla.org/try/rev/af7b179f8277e1c29ef27ee5a49ef9950ae29664
• Push result: https://treeherder.mozilla.org/jobs?repo=try&revision=af7b179f8277e1c29ef27ee5a49ef9950ae29664&selectedTaskRun=ZxKyPwPsTMKsQtD3optCpg.0
• Input path: toolkit/components/extensions/test/marionette (from mach try fuzzy)
• Expected matched manifest: toolkit/components/extensions/test/marionette/manifest-serviceworker.toml
• Actual path passed to runtest.py: /builds/worker/workspace/build/tests/marionette/tests/toolkit/components/extensions/test/marionette

Example from ./mach try --preset=webextensions (example of try push):

• Input path: toolkit/components/extensions/test
• Expected matched manifest: toolkit/components/extensions/test/marionette/manifest-serviceworker.toml
• Actual path passed to runtests.py: /builds/worker/workspace/build/tests/marionette/tests/toolkit/components/extensions/test

Relevant part of log file:

Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/marionette/harness/marionette_harness/runtests.py', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--setpref=layers.d3d11.enable-blacklist=false', '--gecko-log=-', '--log-raw=-', '-vv', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/marionette_errorsummary.log', '--log-html=/builds/worker/workspace/build/blobber_upload_dir/report.html', '--binary=/builds/worker/workspace/build/application/firefox/firefox', '--address=localhost:2828', '--symbols-path=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Cj489SrlTSOF9duuQ0_dJw/artifacts/public/build/target.crashreporter-symbols.zip', '/builds/worker/workspace/build/tests/marionette/tests/toolkit/components/extensions/test'] with output_timeout 1000
INFO - usage: runtests.py [options] test_file_or_dir <test_file_or_dir> ...
INFO - runtests.py: error: Test file(s) not found: /builds/worker/workspace/build/tests/marionette/tests/toolkit/components/extensions/test
INFO - Return code: 2
ERROR - No suite end message was emitted by this harness.

Comment 1

[Rob Wu [:robwu]]

webextensions try preset was added in bug 1856473.

Due to this bug, marionette tests are not scheduled and each try push has orange jobs.
While I could explicitly filter out marionette tests from the try preset (to avoid orange jobs), my preference is to fix this bug so that marionette tests are scheduled as expected.

Comment 2

[Geoff Brown [:gbrown]]

I'm not sure if there is a problem with the path manipulations.

It looks to me like the expected test files are not present in the test archive. afaik, the marionette test files come from the target.common.tests archive. For example, in https://firefoxci.taskcluster-artifacts.net/Ct6-uIRhQdK52Gy3DGPd9A/0/public/logs/live_backing.log:

[task 2024-04-17T07:12:22.973Z] 07:12:22     INFO - Downloading and extracting to /builds/worker/workspace/build/tests these dirs * from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Cj489SrlTSOF9duuQ0_dJw/artifacts/public/build/target.common.tests.tar.gz

but if I examine that target.common.tests.tar.gz file, there are no toolkit/components/extensions paths present.

I wonder if that's related to the conditional at https://searchfox.org/mozilla-central/rev/ee9fd5e2df79c6d69af5aa9bc36041166f483227/toolkit/components/extensions/moz.build#133 ?

Comment 3

[Geoff Brown [:gbrown]]

Actually, I think target.common.tests.tar.gz is normally populated with marionette test files via https://searchfox.org/mozilla-central/rev/ee9fd5e2df79c6d69af5aa9bc36041166f483227/python/mozbuild/mozbuild/action/test_archive.py#108-117 -- that is, from the main manifest, at https://searchfox.org/mozilla-central/rev/ee9fd5e2df79c6d69af5aa9bc36041166f483227/testing/marionette/harness/marionette_harness/tests/unit-tests.toml ... which seems to be missing toolkit/components/extensions/test.

Comment 4

[Geoff Brown [:gbrown]]

Yes, those test tasks pass on try if the missing manifest is added to unit-tests.toml: https://treeherder.mozilla.org/jobs?repo=try&revision=6dc21c9a9b151e2f927d7dccd7f5fbea7cc68a8a

Comment 5

[Geoff Brown [:gbrown]]

Attachment: Bug 1892038 - Add missing marionette manifest path

Comment 6

[Rob Wu [:robwu]]

Bug 1893223 was filed when a concern was raised about the general case of tests being in MARIONETTE_MANIFESTS but not in unit-tests.toml.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11ef59138be1
Add missing marionette manifest path r=webdriver-reviewers,jdescottes,robwu

Comment 8

[tszentpeteri]

https://hg.mozilla.org/mozilla-central/rev/11ef59138be1