1858155 - Attempting to set test_paths to only one test causes an error when the test runs

Status: NEW

(Testing :: Mochitest, defect, P2)

Description

[Greg Mierzwinski [:sparky]]

I noticed an issue where when the test_paths option in Mochitest is set to 1 test. It's caused by something in this code: https://searchfox.org/mozilla-central/source/testing/mochitest/runtests.py#1215

When I only set one test_path there, it tries to navigate to the files directly with a URL like this: http://mochi.test:8888/tests/test_sw_register_performance.html?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&ignorePrefsFile=ignorePrefs.json

It should be navigating to this link though: http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&consoleLevel=INFO&manifestFile=tests.json&dumpOutputDirectory=%2Ftmp&ignorePrefsFile=ignorePrefs.json

If I set the test_paths to at least 2 tests, then it works. I managed to hack around this issue by setting the test_paths to the same test twice. It only runs the test once though.

Here's a patch that can help to reproduce this issue locally: https://hg.mozilla.org/try/rev/8147781df024fe6e5c93a79411085ab76b6a8006#l5.111

You'll then need to run: python3 python/mozperftest/mozperftest/runner.py dom/serviceworkers/test/test_sw_register_performance.html --flavor mochitest --output ./artifacts

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jmaher, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I missed this, thanks for filing it. Mochitest harness is designed that if you pass in a single test, it will run in a different mode, it will ignore skip-if, and other harness stuff as it assumes you are running it locally to debug or test specific things.

when you say test_paths, is this MOZHARNESS_TEST_PATHS?

Comment 3

[Greg Mierzwinski [:sparky]]

Ah good to know, thanks! I'm referring to the test_paths argument here (I'm not sure if it sets MOZHARNESS_TEST_PATHS): https://searchfox.org/mozilla-central/source/python/mozperftest/mozperftest/test/mochitest.py#153

It gets passed to the runtests.run_test_harness here: https://searchfox.org/mozilla-central/source/python/mozperftest/mozperftest/test/mochitest.py#170

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

is this getting resolved via our chat in Slack? using keep_open?