Closed Bug 924977 Opened 7 years ago Closed 7 years ago

MachCommandBase doesn't detect objdir outside of topsrcdir with no mozconfig

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla27

People

(Reporter: froydnj, Assigned: gps)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Using home-grown |configure && make| build system, I needed to run an xpcshell test:

[froydnj@cerebro build-mc]$ ~/src/mozilla-central-official/mach xpcshell-test toolkit/components/telemetry/tests/unit/test_TelemetryPing.js
make: *** No rule to make target `install-tests'.  Stop.
Error running mach:

    ['xpcshell-test', 'toolkit/components/telemetry/tests/unit/test_TelemetryPing.js']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code: [u'/usr/bin/make', u'-j16', u'-s', u'install-tests']

  File "/home/froydnj/src/mozilla-central-official/testing/xpcshell/mach_commands.py", line 237, in run_xpcshell_test
    driver.install_tests(remove=False)
  File "/home/froydnj/src/mozilla-central-official/python/mozbuild/mozbuild/controller/building.py", line 548, in install_tests
    print_directory=False)
  File "/home/froydnj/src/mozilla-central-official/python/mozbuild/mozbuild/base.py", line 456, in _run_make
    return fn(**params)
  File "/home/froydnj/src/mozilla-central-official/python/mozbuild/mozbuild/base.py", line 480, in _run_command_in_objdir
    return self.run_process(cwd=self.topobjdir, **args)
  File "/home/froydnj/src/mozilla-central-official/python/mach/mach/mixin/process.py", line 137, in run_process
    raise Exception('Process executed with non-0 exit code: %s' % args)

Running |make install-tests| from the objdir works:

[froydnj@cerebro build-mc]$ make install-tests
make: Circular /home/froydnj/src/mozilla-central-official/CLOBBER <- /home/froydnj/src/mozilla-central-official/CLOBBER dependency dropped.
MOZBUILD_BACKEND_CHECKED= make -C js/src backend.RecursiveMakeBackend.built
make[1]: Entering directory `/opt/build/froydnj/build-mc/js/src'
make[1]: `backend.RecursiveMakeBackend.built' is up to date.
make[1]: Leaving directory `/opt/build/froydnj/build-mc/js/src'
/opt/build/froydnj/build-mc/_virtualenv/bin/python -m mozbuild.action.process_install_manifest _tests _build_manifests/install/tests js/src/_build_manifests/install/tests
From _tests: Kept 11264 existing; Added/updated 0; Removed 4335 files and 422 directories.

Happy to help with further investigation, this seems really unfortunate.
It's likely mach isn't picking up the objdir properly. Does |mach environment| report the proper paths? If not, please report as many details about your paths, mozconfig, etc. Tell us exactly where it is failing. What revision of the tree are you on? We landed some changes to environment detection in the last week or so that may have regressed things for you.
Blocks: 912114
Flags: needinfo?(nfroyd)
(In reply to Gregory Szorc [:gps] from comment #1)
> It's likely mach isn't picking up the objdir properly. Does |mach
> environment| report the proper paths? If not, please report as many details
> about your paths, mozconfig, etc. Tell us exactly where it is failing. What
> revision of the tree are you on? We landed some changes to environment
> detection in the last week or so that may have regressed things for you.

From within my objdir:

[froydnj@cerebro build-mc]$ ~/src/mozilla-central-official/mach environment
platform:
	Linux-3.2.0-4-amd64-x86_64-with-debian-7.1
python version:
	2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2]
python prefix:
	/usr
mach cwd:
	/home/froydnj/src/mozilla-central-official
os cwd:
	/opt/build/froydnj/build-mc
mach directory:
	/home/froydnj/src/mozilla-central-official
state directory:
	/home/froydnj/.mozbuild
mozconfig path:
	None
object directory:
	/home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu
mozconfig configure args:
mozconfig extra make args:
mozconfig make flags:

I have no mozconfig...is that part of the problem here?  I'm not building with mach, but it seemed like the most reasonable way to run the tests.
Flags: needinfo?(nfroyd)
os cwd:
	/opt/build/froydnj/build-mc
object directory:
	/home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu

That seems to be the problem (unless there is symlink funkiness happening here).

Can you try installing mach into $PATH and running as simply |mach environment|?

https://developer.mozilla.org/en-US/docs/Developer_Guide/mach#Adding_mach_to_your_shell%27s_search_path
Flags: needinfo?(nfroyd)
(In reply to Gregory Szorc [:gps] from comment #3)
> os cwd:
> 	/opt/build/froydnj/build-mc
> object directory:
> 	/home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu
> 
> That seems to be the problem (unless there is symlink funkiness happening
> here).
> 
> Can you try installing mach into $PATH and running as simply |mach
> environment|?
> 
> https://developer.mozilla.org/en-US/docs/Developer_Guide/
> mach#Adding_mach_to_your_shell%27s_search_path

[froydnj@cerebro build-mc]$ cp ~/src/mozilla-central-official/mach ~/bin/
[froydnj@cerebro build-mc]$ mach environment
platform:
	Linux-3.2.0-4-amd64-x86_64-with-debian-7.1
python version:
	2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2]
python prefix:
	/usr
mach cwd:
	/home/froydnj/src/mozilla-central-official
os cwd:
	/opt/build/froydnj/build-mc
mach directory:
	/home/froydnj/src/mozilla-central-official
state directory:
	/home/froydnj/.mozbuild
mozconfig path:
	None
object directory:
	/home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu
mozconfig configure args:
mozconfig extra make args:
mozconfig make flags:

That seems very unexpected.  ~/src/mozilla-central-official is not on my $PATH, ~/bin is, and there are no symlinks floating around here.
Flags: needinfo?(nfroyd)
Also, contents of mozinfo.json:

{"bin_suffix": "", "tests_enabled": true, "appname": "firefox", "os": "linux", "topsrcdir": "/home/froydnj/src/mozilla-central-official", "toolkit": "gtk2", "crashreporter": true, "debug": false, "asan": false, "bits": 64, "processor": "x86_64"}
Can you add |import pdb; pdb.set_trace()| in python/mozbuild/mozbuild/base.py in MachCommandBase.__init__ and let me know what's going on? I suspect the recently added "except BuildEnvironmentNotFoundException" block is getting handled for some esoteric reason.

I should also be able to reproduce this if you give me enough info to recreate your environment (exact configure commands).
Flags: needinfo?(nfroyd)
set_trace dropped me into a debugger, where I didn't want to waste time figuring out what to do, so configury instead:

1. Create new objdir outside topsrcdir.
2. Run

rm -rf *; ${topsrcdir}/configure --enable-optimize --disable-debug --enable-debug-symbols --disable-gstreamer --without-intl-api --disable-dmd; time make -srj20

(adjust make -j to taste)

3. Run |mach environment|.
4. ????
5. Land bugfix.
Flags: needinfo?(nfroyd)
I'm able to reproduce this. Create an objdir outside of topsrcdir with no mozconfig and mach will deduce the objdir using config.guess.
Status: NEW → ASSIGNED
Summary: mach xpcshell-test $PATH complains about install-tests not being a valid target → MachCommandBase doesn't detect objdir outside of topsrcdir with no mozconfig
Hmmm. mach environment doesn't use MachCommandBase. Gah.
Bug is isolated to MachCommandBase passing in cwd=context.cwd which is set to topsrcdir by mach. Now why is it setting cwd to topsrcdir and not os.getcwd(). Hmm.
This patch fixes a lot of fail.

The mach context object was storing "topdir" and "cwd" as the same
thing. This was just plain wrong. "cwd" is now properly reported as the
OS's current working directory.

Also, "topdir" is an invention of the Mozilla-specific parts of mach, so
I moved it to the custom context populator in mach_bootstrap.py.

Since I was touching the base mach file, I decided to fix a long-time
wart of not having everything defined in functions or protected behind
if __name__ == '__main__'.

I also fixed |mach environment| to use MachCommandBase, which it should
have from the beginning.

Finally, I fixed a bug where samepath() was raising when it attempted to
compare paths that didn't exist. Derp.

There's a lot in this patch, I know. There's a high risk random mach
commands will stop working because they relied on previous behavior of
context.cwd. We should be able to fix them all easily enough.
Attachment #816007 - Flags: review?(ahalberstadt)
Assignee: nobody → gps
Comment on attachment 816007 [details] [diff] [review]
More proper current working directory handling in mach

Please confirm this patch fixes things for you. Please note it changes the mach script, so if you copied (not symlinked) to $PATH, you'll need to recopy. (Pretty sure we have a bug somewhere for mach to perform a content validation check if the installed vs discovery versions vary.)
Attachment #816007 - Flags: feedback?(nfroyd)
Comment on attachment 816007 [details] [diff] [review]
More proper current working directory handling in mach

Review of attachment 816007 [details] [diff] [review]:
-----------------------------------------------------------------

See also: https://bugzilla.mozilla.org/show_bug.cgi?id=902581#c8

I hit this same problem originally when getting b2g to work since objdir-gecko was created outside the topsrcdir (though at the time it had a different failure pattern). The b2g bootstrap script also passes in topdir from the populate_context_handler.

::: build/mach_bootstrap.py
@@ +176,2 @@
>  
> +    mach = mach.main.Mach(os.getcwd())

Should the b2g bootstrap script be updated as well? https://github.com/mozilla-b2g/B2G/blob/master/tools/mach_b2g_bootstrap.py#L216

(Note: that b2g_home is neither necessarily topsrcdir nor os.getcwd())
Attachment #816007 - Flags: review?(ahalberstadt) → review+
Comment on attachment 816007 [details] [diff] [review]
More proper current working directory handling in mach

Review of attachment 816007 [details] [diff] [review]:
-----------------------------------------------------------------

Missed a nit.

::: python/mozbuild/mozbuild/mach_commands.py
@@ +852,5 @@
>  
> +        try:
> +            mb = MozbuildObject.from_environment(cwd=self._mach_context.cwd)
> +        except ObjdirMismatchException as e:
> +            print('Amibiguous object directory detected. We detected that '

nit: Ambiguous
https://hg.mozilla.org/mozilla-central/rev/c1c61b305ce0
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla27
Comment on attachment 816007 [details] [diff] [review]
More proper current working directory handling in mach

Seems to work better for me.  Thanks for the fix!
Attachment #816007 - Flags: feedback?(nfroyd) → feedback+
I got bit by this building b2g from mozilla-central this week; thanks for the extensive discussion which helped me get past it with a simple:

  export MOZCONFIG=/full/path/to/.mozconfig
(In reply to Sam Penrose from comment #18)
> I got bit by this building b2g from mozilla-central this week; thanks for
> the extensive discussion which helped me get past it with a simple:
> 
>   export MOZCONFIG=/full/path/to/.mozconfig

Assuming you had a relative path to your MOZCONFIG previously, and that was busted, you might find bug 936555 useful, too.
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.