1197541 - Add --dump-tests option in mochitest/xpcshell for all tests that will be run

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Vaibhav (:vaibhav1994)]

From https://bugzilla.mozilla.org/show_bug.cgi?id=1197318#c4:

ahal:
I'd propose adding a --dump-tests option to mochitest. When this option is passed in, at the end of mochitest.active_tests, instead of returning, it would save the tests that would be run to a file and immediately call sys.exit(). This way, chunk-finder could invoke the harness with whatever command line is being used in automation, with the addition of --dump-tests. It could then scan the files for the missing test. This has the benefit of always being up to date with the right configuration.

You could similarly add a --dump-tests option to xpcshell as well. As another bonus, this approach could work with test harnesses that don't use manifestparser, like reftest.

Comment 1

[Vaibhav (:vaibhav1994)]

Attachment: MozReview Request: Bug 1197541 - Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Comment 2

[Vaibhav (:vaibhav1994)]

Andrew, in the review active_tests.json will be dumped to:
* /{obj_dir}/_tests/testing/mochitest/active_tests.json for mochitests
* /testing/xpcshell/active_tests.json for xpcshell (since there is no self.topobjdir present in it)

Let me know if the paths will be fine or if I need to change them.

Comment 3

[Andrew Halberstadt [:ahal]]

Comment on attachment 8651437 [details]
MozReview Request: Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

https://reviewboard.mozilla.org/r/16945/#review15159

Thanks, looks good. Just some minor comments to address.

::: testing/mochitest/mochitest_options.py:327
(Diff revision 1)
> +          "help": "Dump all the tests that will be run to active_tests.json",

I think we should add "suppress": True here. Alone it isn't that useful for finding chunks because users still need to dig through the file. So might as well hide it from the help menu.

::: testing/mochitest/runtests.py:1982
(Diff revision 1)
> +            active_tests_path = os.path.join(SCRIPT_DIR, 'active_tests.json')

Instead of hardcoding a path, let's just require the path on the command line. That way you can invoke the harness with --dump-tests multiple times without overwriting the file.

Comment 4

[Vaibhav (:vaibhav1994)]

Comment on attachment 8651437 [details]
MozReview Request: Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Comment 5

[Vaibhav (:vaibhav1994)]

Andrew, I have also added --dump-tests to mach for xpcshell-test since now it accepts filename, and it wouldn't make sense if we can't pass it through mach.

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 8651437 [details]
MozReview Request: Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

https://reviewboard.mozilla.org/r/16945/#review15271

A general nit throughout:
Please update from camelcase to underscores for new variable names. I know it's already a mix of both, but new variables should be underscored.

But with that fixed, looks good!

::: testing/mochitest/runtests.py:1982
(Diff revision 2)
> +            # Allow user to use shorthand '~' to home directory

nit: this comment isn't necessary

::: testing/xpcshell/runxpcshelltests.py:832
(Diff revision 2)
> +            # Allow user to use shorthand '~' to home directory

nit: comment not necessary

Comment 7

[Vaibhav (:vaibhav1994)]

Comment on attachment 8651437 [details]
MozReview Request: Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Comment 8

[Vaibhav (:vaibhav1994)]

Comment on attachment 8651437 [details]
MozReview Request: Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Comment 9

[Vaibhav (:vaibhav1994)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=36a67897f18a

Comment 10

[Vaibhav (:vaibhav1994)]

Comment on attachment 8651437 [details]
MozReview Request: Bug 1197541 -  Added --dump-tests option to mochitest and xpcshell for all tests that will be run. r=ahal

Carrying forward the r+

Comment 11

[Vaibhav (:vaibhav1994)]

Try run passes (even though this patch does not require a try run).

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/bb27412c9a79

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/bb27412c9a79