1197543 - Add an option in mochitest to filter the tests according to a given mozinfo

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Vaibhav (:vaibhav1994)]

For chunk-finder, we want to give developers the option to filter and dump tests as they will be for a particular platform (mozinfo) in production. This option can be restricted to be used alongside --dump-tests flag for time being.

Comment 1

[Vaibhav (:vaibhav1994)]

Attachment: MozReview Request: Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

Bug 1197543 - Adding --filter-tests-mozinfo option in mochitest to filter tests for a given mozinfo.

Comment 2

[Vaibhav (:vaibhav1994)]

https://reviewboard.mozilla.org/r/16947/#review15031

::: testing/mochitest/mochitest_options.py:567
(Diff revision 1)
>          if options.totalChunks:

We can add a condition here to ensure that --filter-tests-mozinfo can be used only along with --dump-tests flag once bug https://bugzilla.mozilla.org/show_bug.cgi?id=1197541 lands.

Comment 3

[Vaibhav (:vaibhav1994)]

Comment on attachment 8651439 [details]
MozReview Request: Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

Andrew, I would like to know your feedback regarding --filter-tests-mozinfo patch.

Comment 4

[Andrew Halberstadt [:ahal]]

Comment on attachment 8651439 [details]
MozReview Request: Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

Let me know if it doesn't work, but if you save a file called 'mozinfo.json' in the same directory (or any parent directory) of the script, mochitest should automatically find it because of this function:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#527

I know that's a little hacky, but I think it's better to just re-use that if it works rather than add another argument.

Comment 5

[Vaibhav (:vaibhav1994)]

While that is a nifty hack, it doesn't work. I am getting the following error:

$ ./mach mochitest -f plain --this-chunk 1 --total-chunks 10 --dump-tests ~/Desktop/test1.txt
Error running mach:

    ['mochitest', '-f', 'plain', '--this-chunk', '1', '--total-chunks', '10', '--dump-tests', '/Users/vagrawal/Desktop/test1.txt']

The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.

Please consider filing a bug against mach by going to the URL:

    https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=mach


If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 2] No such file or directory: '/builds/slave/m-in-lx-0000000000000000000000/build/src'

  File "/Users/vagrawal/inbound/python/mach/mach/main.py", line 336, in run
    return self._run(argv)
  File "/Users/vagrawal/inbound/python/mach/mach/main.py", line 382, in _run
    args = parser.parse_args(argv)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1701, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1733, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1921, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1898, in consume_positionals
    take_action(action, args)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/argparse.py", line 1807, in take_action
    action(self, namespace, argument_values, option_string)
  File "/Users/vagrawal/inbound/python/mach/mach/dispatcher.py", line 185, in __call__
    if handler.parser:
  File "/Users/vagrawal/inbound/python/mach/mach/decorators.py", line 77, in parser
    self._parser = self._parser()
  File "/Users/vagrawal/inbound/testing/mochitest/mach_commands.py", line 412, in setup_argument_parser
    ('.py', 'r', imp.PY_SOURCE))
  File "/Users/vagrawal/inbound/obj-x86_64-apple-darwin14.3.0/_tests/testing/mochitest/runtests.py", line 48, in <module>
    from mochitest_options import MochitestArgumentParser, build_obj
  File "/Users/vagrawal/inbound/build/mach_bootstrap.py", line 356, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/Users/vagrawal/inbound/obj-x86_64-apple-darwin14.3.0/_tests/testing/mochitest/mochitest_options.py", line 25, in <module>
    build_obj = MozbuildObject.from_environment(cwd=here)
  File "/Users/vagrawal/inbound/python/mozbuild/mozbuild/base.py", line 169, in from_environment
    config = loader.read_mozconfig(mozconfig, moz_build_app=current_project)
  File "/Users/vagrawal/inbound/python/mozbuild/mozbuild/mozconfig.py", line 232, in read_mozconfig
    cwd=self.topsrcdir, env=env)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.10_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception


:ahal, thoughts?

Comment 6

[Andrew Halberstadt [:ahal]]

Strange, I guess it's because you're overwriting the mozinfo.json in the objdir and it can't find a mozconfig anymore?

Here's another idea: mozinfo is a global dict, so your tool could just import mozinfo and update it manually without passing it in via the command line to mochitest. If that doesn't work and you need to pass it in, please call the argument --extra-mozinfo-json and make it suppress: True.

Comment 7

[Vaibhav (:vaibhav1994)]

The second idea of updating the mozinfo before calling mochitest harness you proposed won't work because we call Mochitest class inside run_test_harness: https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#2588 and it would override the update to mozinfo because of call to update_mozinfo method: https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#527

So I would have to add --extra-mozinfo-json argument.

Comment 8

[Andrew Halberstadt [:ahal]]

Right, ok I guess we'll have to do that then! Sorry for sending you in circles there. Please change the name of the argument, use underscores instead of camelcase and re-flag me for review. Thanks!

Comment 9

[Vaibhav (:vaibhav1994)]

Comment on attachment 8651439 [details]
MozReview Request: Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

Comment 10

[Andrew Halberstadt [:ahal]]

Comment on attachment 8651439 [details]
MozReview Request: Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

https://reviewboard.mozilla.org/r/16949/#review15591

::: testing/mochitest/mochitest_options.py:562
(Diff revision 2)
> +            if not os.path.isfile(mozInfoFile):

mozInfoFile is potentially undefined here. I think you meant to use elif and options.extra_mozinfo_json instead. Also underscores :)

::: testing/mochitest/mochitest_options.py:564
(Diff revision 2)
> +                                Perhaps you need to use --extra-mozinfo-json?" % mozInfoFile)

The error message doesn't really make sense because you can only get it *if* you used --extra-mozinfo-json

Comment 11

[Vaibhav (:vaibhav1994)]

Comment on attachment 8651439 [details]
MozReview Request: Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

Comment 12

[Vaibhav (:vaibhav1994)]

Updated. Those lines were really not required, I copied from other part of codebase, should have removed them :)

Comment 13

[Andrew Halberstadt [:ahal]]

Comment on attachment 8651439 [details]
MozReview Request: Bug 1197543 - Adding --extra-mozinfo-json option in mochitest to filter tests for a given mozinfo file.r=ahal

https://reviewboard.mozilla.org/r/16949/#review15617

Thanks, looks good to me! p.s If you go to the review page, you can mark the issues closed and reviewboard will give you a nice green checkmark.

Comment 14

[Vaibhav (:vaibhav1994)]

Thanks Andrew!

Comment 15

[Vaibhav (:vaibhav1994)]

This patch does not need a try push, as it adds an option and is mach build work locally for me.

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/cb7e5c02a0a8

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/cb7e5c02a0a8