Closed Bug 899849 Opened 6 years ago Closed 6 years ago

Properly resolve build environment when running inside virtualenv

Categories

(Firefox Build System :: General, defect, blocker)

x86
macOS
defect
Not set
blocker

Tracking

(Not tracked)

RESOLVED FIXED
mozilla25

People

(Reporter: bzbarsky, Assigned: ted)

References

Details

Attachments

(1 file)

CHANGESET: d879c9cff194

STEPS TO REPRODUCE:

mozilla% ../obj-firefox/_virtualenv/bin/python ../obj-firefox/_tests/testing/mochitest/runtests.py --test-path=content/base/test/test_nodelist_holes.html

EXPECTED RESULTS: run the test

ACTUAL RESULTS:

Traceback (most recent call last):
  File "../obj-firefox/_tests/testing/mochitest/runtests.py", line 25, in <module>
    from mochitest_options import MochitestOptions
  File "/Users/bzbarsky/mozilla/vanilla/obj-firefox/_tests/testing/mochitest/mochitest_options.py", line 27, in <module>
    class MochitestOptions(optparse.OptionParser):
  File "/Users/bzbarsky/mozilla/vanilla/obj-firefox/_tests/testing/mochitest/mochitest_options.py", line 48, in MochitestOptions
    "default": build_obj.get_binary_path() if build_obj is not None else None,
  File "/Users/bzbarsky/mozilla/vanilla/mozilla/python/mozbuild/mozbuild/base.py", line 263, in get_binary_path
    substs = self.substs
  File "/Users/bzbarsky/mozilla/vanilla/mozilla/python/mozbuild/mozbuild/base.py", line 223, in substs
    return self.config_environment.substs
  File "/Users/bzbarsky/mozilla/vanilla/mozilla/python/mozbuild/mozbuild/base.py", line 210, in config_environment
    raise Exception('config.status not available. Run configure.')
Exception: config.status not available. Run configure.

and this happens because the scripts are looking for config.status in the default objdir (which doesn't even exist!), not in the actual objdir involved.

Looks like a regression from bug 865349, as far as I can tell based on who's invoking the (old) bogus code.

Hard to write patches if I can't test them....
Do you get the same error running via mach or the old make targets?
No idea about mach, but doing this:

mozilla% make TEST_PATH=content/base/test/test_nodelist_holes.html -C ../obj-firefox mochitest-plain                                           

works correctly.
you can add --xre-path, --utility-path, --certificate-path to the commandline and it should work.
Do you have the $MOZCONFIG environment set? It's my understanding that mozbuild will try to find the objdir based on the currently active mozconfig (or srcdir/mozconfig if none is set). I think mach has a check that will error out if you operate from a different objdir than the currently active mozconfig, but if you call runtests.py directly this check is bypassed.

I'm not sure if I'm 100% correct here, but if I am perhaps that check should be moved out of mach and into mozbuild.
> Do you have the $MOZCONFIG environment set?

No, because I have multiple objdirs depending on what I'm doing.  Running tests from a specific objdir without having to enter it, and being able to very quickly switch which objdir I'm using right now is pretty important.
Currently the objdir is found based on the mozconfig, the mozconfig used follows these rules:
http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/mozconfig.py#78

It sounds like instead of finding the objdir from the mozconfig, if we detect that the cwd is an objdir, we should do it the other way around (i.e use the objdir to find the mozconfig). I think we can use the mozconfig key in config.status/mozinfo.json.

Gps, make sense?
mozbuild definitely wants to work in this situation:
http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#124

It tries to see if you're running a Python script from the objdir, and also looks to see if the Python you're using is in the objdir virtualenv. Clearly something here isn't working right.
Ah! http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/base.py#124

bz is running the script from outside the objdir, so the mozinfo.json isn't detected and we default to the currently active one. I think ideally we want to replace os.getcwd() with the dirname of the caller of from_environment().

Boris, does it work if you cd to the objdir and run from there?
This is dumb. If you're running from within the srcdir, the current check in from_environment will find that and then skip the "am I running from a virtualenv Python" check, and then give you a useless MozbuildObject.
Attachment #783895 - Flags: review?(gps)
Assignee: nobody → ted
Comment on attachment 783895 [details] [diff] [review]
fix MozbuildObject.from_environment to work right when no mozconfig specified

Review of attachment 783895 [details] [diff] [review]:
-----------------------------------------------------------------

Makes sense. I wonder whose workflow this will subtly break. Every change to resolving the current environment seemingly impacts some small market.
Attachment #783895 - Flags: review?(gps) → review+
Component: Mochitest → Build Config
Product: Testing → Core
Summary: Unable to run mochitests using runtests.py → Properly resolve build environment when running inside virtualenv
> Boris, does it work if you cd to the objdir and run from there?

Fwiw, yes.
https://hg.mozilla.org/mozilla-central/rev/0feed139b856
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla25
Blocks: 907552
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.