1164003 - mach mochitest is looking for runtests.py in the wrong place

Status: RESOLVED WONTFIX

(Firefox Build System :: Mach Core, defect)

Description

[Ian Neal]

mach build and mach run work fine against SeaMonkey, but when you try and run mach mochitest you get the following error:
Error running mach:

    ['mochitest']

The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.

Please consider filing a bug against mach by going to the URL:

    https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=mach


If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

IOError: [Errno 2] No such file or directory: u'/home/gizmo/central/comm-central/mozilla/obj-x86_64-unknown-linux-gnu/_tests/testing/mochitest/runtests.py'

  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/main.py", line 344, in run
    return self._run(argv)
  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/main.py", line 390, in _run
    args = parser.parse_args(argv)
  File "/usr/lib64/python2.7/argparse.py", line 1694, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib64/python2.7/argparse.py", line 1726, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib64/python2.7/argparse.py", line 1935, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib64/python2.7/argparse.py", line 1891, in consume_positionals
    take_action(action, args)
  File "/usr/lib64/python2.7/argparse.py", line 1800, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/dispatcher.py", line 172, in __call__
    if handler.parser:
  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/base.py", line 123, in parser
    self._parser = self._parser()
  File "/home/gizmo/central/comm-central/mozilla/testing/mochitest/mach_commands.py", line 351, in setup_argument_parser
    with open(path, 'r') as fh:

Not sure why it is trying to look in /home/gizmo/central/comm-central/mozilla/obj-x86_64-unknown-linux-gnu/_tests/testing/mochitest/ and .mozconfig points the OBJDIR to /home/gizmo/central/sm-opt and runtests.py lives under /home/gizmo/central/sm-opt/_tests/testing/mochitest/runtests.py

This is a recent regression as far as I can tell.

Comment 1

[Gregory Szorc [:gps]]

This might be fallout from ahal's recent refactorings.

Comment 2

[Andrew Halberstadt [:ahal]]

Looks like it fell back to config.guess for the objdir which means MozbuildObject.from_environment couldn't find it:
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/mach_commands.py#311

Sorry I'm not familiar with seamonkey. Normally the above works because 'here' == the location of mach_commands.py which is in the topsrcdir (so it should find the .mozconfig and subsequently the objdir). Can you think of any reasons why that might not be the case for you?

Comment 3

[Andrew Halberstadt [:ahal]]

p.s I have a feeling a workaround for you might be:

export MOZCONFIG=path/to/.mozconfig

Comment 4

[Ian Neal]

Only thing I can think is that under comm-central the mach command would sit in the mozilla subdirectory of the source directory, whereas the .mozconfig is in the root of the source directory.

mach build seems to be able to find the .mozconfig

It doesn't fall back in the following two circumstances:
1/ If MOZCONFIG environment variable is set and points to the .mozconfig in the root of the source directory.
2/ If MOZCONFIG environment variable is not set, but the .mozconfig is copied from the root of the source directory to the mozilla subdirectory.

Comment 5

[Andrew Halberstadt [:ahal]]

(In reply to Ian Neal from comment #4)
> Only thing I can think is that under comm-central the mach command would sit
> in the mozilla subdirectory of the source directory, whereas the .mozconfig
> is in the root of the source directory.

Yeah that would do it. MozbuildObject.from_environment() stops searching for the mozconfig once it reaches topsrcdir:
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#148

So the question is how come it used to work before my change, as far as I can tell MachCommandBase instances use MozbuildObject.from_environment() the same way:
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#646

Comment 6

[Ian Neal]

Which is the change you are referring to so I can back out locally and see if it fixes the issue I see?

Comment 7

[Andrew Halberstadt [:ahal]]

You can try, but I think you'll get conflicts if you back it out as there have been a series of patches landed all depending on one another. I'll dig into this a bit.

For now I'd recommend exporting MOZCONFIG in your .bashrc or equivalent as a workaround.

Comment 8

[Andrew Halberstadt [:ahal]]

Ian, what happens if you replace 'here' with 'os.getcwd()'?
https://dxr.mozilla.org/mozilla-central/source/testing/mochitest/mach_commands.py#338

(this is just a guess)

Comment 9

[Ian Neal]

Slightly different error message:
BuildEnvironmentNotFoundException: Could not find Mozilla source tree or build environment.

  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/main.py", line 336, in run
    return self._run(argv)
  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/main.py", line 382, in _run
    args = parser.parse_args(argv)
  File "/usr/lib64/python2.7/argparse.py", line 1694, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib64/python2.7/argparse.py", line 1726, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib64/python2.7/argparse.py", line 1935, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib64/python2.7/argparse.py", line 1891, in consume_positionals
    take_action(action, args)
  File "/usr/lib64/python2.7/argparse.py", line 1800, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/dispatcher.py", line 172, in __call__
    if handler.parser:
  File "/home/gizmo/central/comm-central/mozilla/python/mach/mach/base.py", line 123, in parser
    self._parser = self._parser()
  File "/home/gizmo/central/comm-central/mozilla/testing/mochitest/mach_commands.py", line 338, in setup_argument_parser
    build_obj = MozbuildObject.from_environment(cwd=os.getcwd())
  File "/home/gizmo/central/comm-central/mozilla/python/mozbuild/mozbuild/base.py", line 160, in from_environment
    'Could not find Mozilla source tree or build environment.')

It does the same if I change it to just "build_obj = MozbuildObject.from_environment()"

Comment 10

[Andrew Halberstadt [:ahal]]

Hmm, so when instantiating the MozbuildObject via the MachCommandBase class, there's this except block:
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#650

That might explain something, but I still don't understand how it's finding a mozconfig outside of the topsrcdir. Are you running the mach command from inside the mozilla objdir?

Comment 11

[Ian Neal]

I am running it from the src directory, so:
./mozilla/mach mochitest
If I start in the obj directory (which is not a subdirectory of the src directory):
../comm-central/mozilla/mach mochitest
I get the same output that I used to get prior to the changes.

Comment 12

[Ian Neal]

If I copy mach from comm-central/mozilla into comm-central, then run from the src directory:
./mach mochitest
that also products the same output that I used to get prior to the changes.
Looks like mach seems to assume that it sits in the topsrcdir rather than checking what the topsrcdir is.

Comment 13

[(not currently active) Ted Mielczarek]

Just for sanity, what does this print?
./mach python -c "from mozbuild.base import MozbuildObject; print MozbuildObject.from_environment().topobjdir"

Comment 14

[Andrew Halberstadt [:ahal]]

(In reply to Ian Neal from comment #12)
> Looks like mach seems to assume that it sits in the topsrcdir rather than
> checking what the topsrcdir is.

Technically in the context of mozbuild/mach, topsrcdir is always the mozilla (gecko) dir. Someone correct me if I'm wrong, but I don't believe mozbuild is aware of comm-central or anything else outside of topsrcdir or topobjdir, which is what makes this tricky.

Comment 15

[Gregory Szorc [:gps]]

mach/mozbuild does work with comm-central. There was a long stream of bugs to get MozbuildObject.from_environment() and friends to play nice with comm-central.

Also, there is a `mach environment` command that may be helpful to debug issues.

Comment 16

[Ian Neal]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #13)
> Just for sanity, what does this print?
> ./mach python -c "from mozbuild.base import MozbuildObject; print
> MozbuildObject.from_environment().topobjdir"

Error running mach:

    ['python', '-c', 'from mozbuild.base import MozbuildObject; print MozbuildObject.from_environment().topobjdir']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 2] No such file or directory: '/home/gizmo/central/comm-central/build/virtualenv_packages.txt'

  File "/home/gizmo/central/comm-central/mozilla/python/mach_commands.py", line 34, in python
    self._activate_virtualenv()
  File "/home/gizmo/central/comm-central/mozilla/python/mozbuild/mozbuild/base.py", line 625, in _activate_virtualenv
    self.virtualenv_manager.ensure()
  File "/home/gizmo/central/comm-central/mozilla/python/mozbuild/mozbuild/virtualenv.py", line 126, in ensure
    if self.up_to_date():
  File "/home/gizmo/central/comm-central/mozilla/python/mozbuild/mozbuild/virtualenv.py", line 98, in up_to_date
    dep_mtime = max(os.path.getmtime(p) for p in deps)
  File "/home/gizmo/central/comm-central/mozilla/python/mozbuild/mozbuild/virtualenv.py", line 98, in <genexpr>
    dep_mtime = max(os.path.getmtime(p) for p in deps)
  File "/usr/lib64/python2.7/genericpath.py", line 54, in getmtime
    return os.stat(filename).st_mtime

Comment 17

[Ian Neal]

(In reply to Gregory Szorc [:gps] from comment #15)
> mach/mozbuild does work with comm-central. There was a long stream of bugs
> to get MozbuildObject.from_environment() and friends to play nice with
> comm-central.
> 
> Also, there is a `mach environment` command that may be helpful to debug
> issues.
Output from running ./mozilla/mach environment command in the source directory is:
platform:
	Linux-3.19.7-200.fc21.x86_64-x86_64-with-fedora-21-Twenty_One
python version:
	2.7.8 (default, Apr 15 2015, 09:26:43) 
[GCC 4.9.2 20150212 (Red Hat 4.9.2-6)]
python prefix:
	/usr
mach cwd:
	/home/gizmo/central/comm-central
os cwd:
	/home/gizmo/central/comm-central
mach directory:
	/home/gizmo/central/comm-central
state directory:
	/home/gizmo/.mozbuild
object directory:
	/home/gizmo/central/sm-opt
mozconfig path:
	/home/gizmo/central/comm-central/.mozconfig
mozconfig configure args:
	--enable-application=suite
	--enable-calendar
	--enable-debugger-info-modules=yes
	--enable-optimizer=-O2 -gstabs+
	--with-ccache=/usr/bin/ccache
mozconfig extra make args:
	MOZ_CO_PROJECT=suite
mozconfig make flags:
	-j4
config topsrcdir:
	/home/gizmo/central/comm-central/mozilla
config topobjdir:
	/home/gizmo/central/sm-opt

Comment 18

[Sylvestre Ledru [:Sylvestre]]

Seems that we won't fix that bug, closing.