1606112 - Attempting to run a single devtools test by modifying tests_list fails

Status: VERIFIED WONTFIX

(Testing :: Mochitest, defect)

Description

[Kyle Huey [:khuey]]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0

Steps to reproduce:

Pernosco patches desktop_unittest.py in DesktopUnittest::_run_category_suites to add the specific test it wants to run to tests_list. This works for most test suites but not for devtools tests.

Actual results:

We see the following output when trying to run just e.g. devtools/client/webconsole/test/browser/browser_webconsole_block_mixedcontent_securityerrors.js:

[task 2019-12-20T00:24:29.522Z] 00:24:29 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtests.py', u'devtools
/client/webconsole/test/browser/browser.ini', '--setpref=fission.autostart=true', '--setpref=dom.serviceWorkers.parent_intercept=true', '--setpref=browser.tabs.documentchannel=true', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--appname=/builds/worker/workspace/build/application/firefox/firefox', '--utility-path=tests
/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=/builds/worker/workspace/build/symbols', '--certificate-path=tests/certs', '--setpref=webgl.force-enabled=true', '--quiet',
'--log-raw=/builds/worker/workspace/build/blobber_upload_dir/mochitest-devtools-chrome_raw.log', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/mochitest-devtools-chr
ome_errorsummary.log', '--use-test-media-devices', '--screenshot-on-fail', '--cleanup-crashes', '--marionette-startup-timeout=180', '--sandbox-read-whitelist=/builds/worker/workspace/build'
, '--log-raw=-', '--flavor=browser', '--subsuite=devtools', '--chunk-by-runtime', '--', 'devtools/client/webconsole/test/browser/browser_webconsole_block_mixedcontent_securityerrors.js'] wi
th output_timeout 1000
[task 2019-12-20T00:24:29.522Z] 00:24:29 INFO - usage: runtests.py [options] [test paths]
[task 2019-12-20T00:24:29.522Z] 00:24:29 INFO - runtests.py: error: unrecognized arguments: -- devtools/client/webconsole/test/browser/browser_webconsole_block_mixedcontent_securityerrors.js

Something does not recognize the -- which is added in query_tests_args (perhaps this code https://searchfox.org/mozilla-central/source/third_party/python/pytest/src/_pytest/config/argparsing.py#336 ?)

Expected results:

The test harness should run the one test that I put in tests_list.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:ahal, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Andrew Halberstadt [:ahal]]

Rather than patching desktop_unittest.py, try setting the MOZHARNESS_TEST_PATHS environment variable. It's how we schedule test paths in our CI so will be well supported. It is a stringified JSON object of the form:

{ <suite>: [<paths>] }

E.g, in your case you'd set it to {"mochitest-devtools-chrome": ["devtools/client/webconsole/test/browser/browser_webconsole_block_mixedcontent_securityerrors.js"]}.

The valid suite names can be found in this variable:
https://searchfox.org/mozilla-central/rev/dd1dafd5c9c05640e76af30b58749076e0199704/testing/mozbase/moztest/moztest/resolve.py#41

Though only suites handled by desktop_unittest.py are supported. Let me know if you need a hand with anything.

Comment 3

[Kyle Huey [:khuey]]

Thanks, that does indeed work.