Closed
Bug 924977
Opened 11 years ago
Closed 11 years ago
MachCommandBase doesn't detect objdir outside of topsrcdir with no mozconfig
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla27
People
(Reporter: froydnj, Assigned: gps)
References
(Blocks 1 open bug)
Details
Attachments
(1 file)
11.05 KB,
patch
|
ahal
:
review+
froydnj
:
feedback+
|
Details | Diff | Splinter Review |
Using home-grown |configure && make| build system, I needed to run an xpcshell test: [froydnj@cerebro build-mc]$ ~/src/mozilla-central-official/mach xpcshell-test toolkit/components/telemetry/tests/unit/test_TelemetryPing.js make: *** No rule to make target `install-tests'. Stop. Error running mach: ['xpcshell-test', 'toolkit/components/telemetry/tests/unit/test_TelemetryPing.js'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You should consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: Exception: Process executed with non-0 exit code: [u'/usr/bin/make', u'-j16', u'-s', u'install-tests'] File "/home/froydnj/src/mozilla-central-official/testing/xpcshell/mach_commands.py", line 237, in run_xpcshell_test driver.install_tests(remove=False) File "/home/froydnj/src/mozilla-central-official/python/mozbuild/mozbuild/controller/building.py", line 548, in install_tests print_directory=False) File "/home/froydnj/src/mozilla-central-official/python/mozbuild/mozbuild/base.py", line 456, in _run_make return fn(**params) File "/home/froydnj/src/mozilla-central-official/python/mozbuild/mozbuild/base.py", line 480, in _run_command_in_objdir return self.run_process(cwd=self.topobjdir, **args) File "/home/froydnj/src/mozilla-central-official/python/mach/mach/mixin/process.py", line 137, in run_process raise Exception('Process executed with non-0 exit code: %s' % args) Running |make install-tests| from the objdir works: [froydnj@cerebro build-mc]$ make install-tests make: Circular /home/froydnj/src/mozilla-central-official/CLOBBER <- /home/froydnj/src/mozilla-central-official/CLOBBER dependency dropped. MOZBUILD_BACKEND_CHECKED= make -C js/src backend.RecursiveMakeBackend.built make[1]: Entering directory `/opt/build/froydnj/build-mc/js/src' make[1]: `backend.RecursiveMakeBackend.built' is up to date. make[1]: Leaving directory `/opt/build/froydnj/build-mc/js/src' /opt/build/froydnj/build-mc/_virtualenv/bin/python -m mozbuild.action.process_install_manifest _tests _build_manifests/install/tests js/src/_build_manifests/install/tests From _tests: Kept 11264 existing; Added/updated 0; Removed 4335 files and 422 directories. Happy to help with further investigation, this seems really unfortunate.
Assignee | ||
Comment 1•11 years ago
|
||
It's likely mach isn't picking up the objdir properly. Does |mach environment| report the proper paths? If not, please report as many details about your paths, mozconfig, etc. Tell us exactly where it is failing. What revision of the tree are you on? We landed some changes to environment detection in the last week or so that may have regressed things for you.
Blocks: 912114
Flags: needinfo?(nfroyd)
Reporter | ||
Comment 2•11 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #1) > It's likely mach isn't picking up the objdir properly. Does |mach > environment| report the proper paths? If not, please report as many details > about your paths, mozconfig, etc. Tell us exactly where it is failing. What > revision of the tree are you on? We landed some changes to environment > detection in the last week or so that may have regressed things for you. From within my objdir: [froydnj@cerebro build-mc]$ ~/src/mozilla-central-official/mach environment platform: Linux-3.2.0-4-amd64-x86_64-with-debian-7.1 python version: 2.7.3 (default, Jan 2 2013, 13:56:14) [GCC 4.7.2] python prefix: /usr mach cwd: /home/froydnj/src/mozilla-central-official os cwd: /opt/build/froydnj/build-mc mach directory: /home/froydnj/src/mozilla-central-official state directory: /home/froydnj/.mozbuild mozconfig path: None object directory: /home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu mozconfig configure args: mozconfig extra make args: mozconfig make flags: I have no mozconfig...is that part of the problem here? I'm not building with mach, but it seemed like the most reasonable way to run the tests.
Flags: needinfo?(nfroyd)
Assignee | ||
Comment 3•11 years ago
|
||
os cwd: /opt/build/froydnj/build-mc object directory: /home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu That seems to be the problem (unless there is symlink funkiness happening here). Can you try installing mach into $PATH and running as simply |mach environment|? https://developer.mozilla.org/en-US/docs/Developer_Guide/mach#Adding_mach_to_your_shell%27s_search_path
Flags: needinfo?(nfroyd)
Reporter | ||
Comment 4•11 years ago
|
||
(In reply to Gregory Szorc [:gps] from comment #3) > os cwd: > /opt/build/froydnj/build-mc > object directory: > /home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu > > That seems to be the problem (unless there is symlink funkiness happening > here). > > Can you try installing mach into $PATH and running as simply |mach > environment|? > > https://developer.mozilla.org/en-US/docs/Developer_Guide/ > mach#Adding_mach_to_your_shell%27s_search_path [froydnj@cerebro build-mc]$ cp ~/src/mozilla-central-official/mach ~/bin/ [froydnj@cerebro build-mc]$ mach environment platform: Linux-3.2.0-4-amd64-x86_64-with-debian-7.1 python version: 2.7.3 (default, Jan 2 2013, 13:56:14) [GCC 4.7.2] python prefix: /usr mach cwd: /home/froydnj/src/mozilla-central-official os cwd: /opt/build/froydnj/build-mc mach directory: /home/froydnj/src/mozilla-central-official state directory: /home/froydnj/.mozbuild mozconfig path: None object directory: /home/froydnj/src/mozilla-central-official/obj-x86_64-unknown-linux-gnu mozconfig configure args: mozconfig extra make args: mozconfig make flags: That seems very unexpected. ~/src/mozilla-central-official is not on my $PATH, ~/bin is, and there are no symlinks floating around here.
Flags: needinfo?(nfroyd)
Reporter | ||
Comment 5•11 years ago
|
||
Also, contents of mozinfo.json: {"bin_suffix": "", "tests_enabled": true, "appname": "firefox", "os": "linux", "topsrcdir": "/home/froydnj/src/mozilla-central-official", "toolkit": "gtk2", "crashreporter": true, "debug": false, "asan": false, "bits": 64, "processor": "x86_64"}
Assignee | ||
Comment 6•11 years ago
|
||
Can you add |import pdb; pdb.set_trace()| in python/mozbuild/mozbuild/base.py in MachCommandBase.__init__ and let me know what's going on? I suspect the recently added "except BuildEnvironmentNotFoundException" block is getting handled for some esoteric reason. I should also be able to reproduce this if you give me enough info to recreate your environment (exact configure commands).
Flags: needinfo?(nfroyd)
Reporter | ||
Comment 7•11 years ago
|
||
set_trace dropped me into a debugger, where I didn't want to waste time figuring out what to do, so configury instead: 1. Create new objdir outside topsrcdir. 2. Run rm -rf *; ${topsrcdir}/configure --enable-optimize --disable-debug --enable-debug-symbols --disable-gstreamer --without-intl-api --disable-dmd; time make -srj20 (adjust make -j to taste) 3. Run |mach environment|. 4. ???? 5. Land bugfix.
Flags: needinfo?(nfroyd)
Assignee | ||
Comment 8•11 years ago
|
||
I'm able to reproduce this. Create an objdir outside of topsrcdir with no mozconfig and mach will deduce the objdir using config.guess.
Status: NEW → ASSIGNED
Summary: mach xpcshell-test $PATH complains about install-tests not being a valid target → MachCommandBase doesn't detect objdir outside of topsrcdir with no mozconfig
Assignee | ||
Comment 9•11 years ago
|
||
Hmmm. mach environment doesn't use MachCommandBase. Gah.
Assignee | ||
Comment 10•11 years ago
|
||
Bug is isolated to MachCommandBase passing in cwd=context.cwd which is set to topsrcdir by mach. Now why is it setting cwd to topsrcdir and not os.getcwd(). Hmm.
Assignee | ||
Comment 11•11 years ago
|
||
This patch fixes a lot of fail. The mach context object was storing "topdir" and "cwd" as the same thing. This was just plain wrong. "cwd" is now properly reported as the OS's current working directory. Also, "topdir" is an invention of the Mozilla-specific parts of mach, so I moved it to the custom context populator in mach_bootstrap.py. Since I was touching the base mach file, I decided to fix a long-time wart of not having everything defined in functions or protected behind if __name__ == '__main__'. I also fixed |mach environment| to use MachCommandBase, which it should have from the beginning. Finally, I fixed a bug where samepath() was raising when it attempted to compare paths that didn't exist. Derp. There's a lot in this patch, I know. There's a high risk random mach commands will stop working because they relied on previous behavior of context.cwd. We should be able to fix them all easily enough.
Attachment #816007 -
Flags: review?(ahalberstadt)
Assignee | ||
Updated•11 years ago
|
Assignee: nobody → gps
Assignee | ||
Comment 12•11 years ago
|
||
Comment on attachment 816007 [details] [diff] [review] More proper current working directory handling in mach Please confirm this patch fixes things for you. Please note it changes the mach script, so if you copied (not symlinked) to $PATH, you'll need to recopy. (Pretty sure we have a bug somewhere for mach to perform a content validation check if the installed vs discovery versions vary.)
Attachment #816007 -
Flags: feedback?(nfroyd)
Comment 13•11 years ago
|
||
Comment on attachment 816007 [details] [diff] [review] More proper current working directory handling in mach Review of attachment 816007 [details] [diff] [review]: ----------------------------------------------------------------- See also: https://bugzilla.mozilla.org/show_bug.cgi?id=902581#c8 I hit this same problem originally when getting b2g to work since objdir-gecko was created outside the topsrcdir (though at the time it had a different failure pattern). The b2g bootstrap script also passes in topdir from the populate_context_handler. ::: build/mach_bootstrap.py @@ +176,2 @@ > > + mach = mach.main.Mach(os.getcwd()) Should the b2g bootstrap script be updated as well? https://github.com/mozilla-b2g/B2G/blob/master/tools/mach_b2g_bootstrap.py#L216 (Note: that b2g_home is neither necessarily topsrcdir nor os.getcwd())
Attachment #816007 -
Flags: review?(ahalberstadt) → review+
Comment 14•11 years ago
|
||
Comment on attachment 816007 [details] [diff] [review] More proper current working directory handling in mach Review of attachment 816007 [details] [diff] [review]: ----------------------------------------------------------------- Missed a nit. ::: python/mozbuild/mozbuild/mach_commands.py @@ +852,5 @@ > > + try: > + mb = MozbuildObject.from_environment(cwd=self._mach_context.cwd) > + except ObjdirMismatchException as e: > + print('Amibiguous object directory detected. We detected that ' nit: Ambiguous
Assignee | ||
Comment 15•11 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/c1c61b305ce0
Flags: in-testsuite-
https://hg.mozilla.org/mozilla-central/rev/c1c61b305ce0
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla27
Reporter | ||
Comment 17•11 years ago
|
||
Comment on attachment 816007 [details] [diff] [review] More proper current working directory handling in mach Seems to work better for me. Thanks for the fix!
Attachment #816007 -
Flags: feedback?(nfroyd) → feedback+
Comment 18•11 years ago
|
||
I got bit by this building b2g from mozilla-central this week; thanks for the extensive discussion which helped me get past it with a simple: export MOZCONFIG=/full/path/to/.mozconfig
Reporter | ||
Comment 19•11 years ago
|
||
(In reply to Sam Penrose from comment #18) > I got bit by this building b2g from mozilla-central this week; thanks for > the extensive discussion which helped me get past it with a simple: > > export MOZCONFIG=/full/path/to/.mozconfig Assuming you had a relative path to your MOZCONFIG previously, and that was busted, you might find bug 936555 useful, too.
Updated•6 years ago
|
Product: Core → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•