855262 - add MozBuildObject.from_environment to easily get build configuration for current objdir

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

bug 648681 added a way to get the default app location from the objdir. It's kind of tied up in some mozbuild APIs though. I would like to have a simpler thing, like:
from mozbuild import get_binary_path
app = get_binary_path()

that we could use to replace automation.py for running tests in the objdir.

Comment 1

[Gregory Szorc [:gps]]

Well, getting a binary path implies figuring out the current tree configuration. That means knowing the objdir, parsing a config.status, ...

At the end of the day, I think we should provide a high-level API for "obtain a handle on the current build environment" and hang "get_binary_path" off of that. e.g.

from mozbuild import BuildSystem

bs = BuildSystem.from_objdir('/path/to/objdir')

OR

# looks at cwd and environment variables, tries to find a mozconfig, etc.
bs = BuildSystem.from_environment()


Then:

bs.get_binary_path()


We can eventually use this class for other things, like:

bs.configure()
bs.build()

If the object gets too unwieldy, we can hang things off of it:

bs.building.configure()
bs.building.generate_backend('visual-studio', version=2012)

That's my vision. We can start by settling on a name and hanging get_binary_path off of it. Sound good?

Comment 2

[(not currently active) Ted Mielczarek]

"BuildSystem" seems fine. I would be ok with some light magic involving the module knowing that it's installed into the virtualenv Python, and locating the objdir root using that knowledge.

Comment 3

[Gregory Szorc [:gps]]

Attachment: BuildSystem class, v1

Initial stab at it. No virtualenv detection yet. I probably also need some sys.path bootstrap magic somewhere for this to really be useful. Will keep iterating.

Comment 4

[(not currently active) Ted Mielczarek]

Attachment: add MozbuildObject.from_environment

Stealing this. At gps' suggestion I moved the from_environment method here to live on mozbuild.base.MozbuildObject, since that provides all the useful methods I need. gps will probably refactor things at some point, but this should work for now.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 752271 [details] [diff] [review]
add MozbuildObject.from_environment

f?gps since this is mostly your code, just want to make sure you like the execution here. I'll get glandium to review it for real.

The only significant change from the previous patch is that I did hook up virtualenv detection--it just looks in the parent directory of sys.prefix for a mozinfo.json.

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 752271 [details] [diff] [review]
add MozbuildObject.from_environment

Review of attachment 752271 [details] [diff] [review]:
-----------------------------------------------------------------

Looks mostly good. I'm having an inner monologue over how much testing, if any, to require. I guess it depends on your intended usage. I could see this function being used for a lot of things (most likely after a copy) and since I imagine a lot of consumers will use this, I'd kinda like to have good test coverage now. That being said, this kind of stuff in the existing build system likely doesn't have robust test coverage, so I dunno.

::: python/mozbuild/mozbuild/base.py
@@ +125,5 @@
> +
> +            # We choose an arbitrary file as an indicator that this is a
> +            # srcdir. We go with ourself because why not!
> +            our_path = os.path.join(dir_path, 'python', 'mozbuild', 'mozbuild',
> +                'manager', 'main.py')

Wrong path.

@@ +132,5 @@
> +                break
> +
> +        if not topsrcdir:
> +            # See if we're running from a Python virtualenv that's inside an objdir.
> +            mozinfo_path = os.path.join(os.path.dirname(sys.prefix), "mozinfo.json")

sys.prefix by itself is a little scary. See http://stackoverflow.com/questions/1871549/python-determine-if-running-inside-virtualenv for slightly better alternative.

@@ +135,5 @@
> +            # See if we're running from a Python virtualenv that's inside an objdir.
> +            mozinfo_path = os.path.join(os.path.dirname(sys.prefix), "mozinfo.json")
> +            if os.path.isfile(mozinfo_path):
> +                topsrcdir, topobjdir, mozconfig = load_mozinfo(mozinfo_path)
> +                print("topsrcdir: %s" % topsrcdir)

print?!

Comment 7

[Gregory Szorc [:gps]]

This patch overlaps significantly with stuff vlad was doing with mach.

Comment 8

[Vladan Djeric (:vladan)]

(In reply to Gregory Szorc [:gps] from comment #7)
> This patch overlaps significantly with stuff vlad was doing with mach.

I think you meant to CC the other Vlad

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Gregory Szorc [:gps] from comment #6)
> Wrong path.

Ooops.

> sys.prefix by itself is a little scary. See
> http://stackoverflow.com/questions/1871549/python-determine-if-running-
> inside-virtualenv for slightly better alternative.

That stackoverflow post is where I got the idea! I don't see anything better suggested there, and the linked PEP indicates that sys.prefix should always be set reliably in a virtualenv.

> print?!

Leftover debugging cruft, sorry.

Comment 10

[(not currently active) Ted Mielczarek]

Attachment: add MozbuildObject.from_environment

Fixed up nits that gps pointed out.

Comment 11

[(not currently active) Ted Mielczarek]

Attachment: add MozbuildObject.from_environment

Oops, I still hadn't fixed that "path to source file" properly.

Comment 12

[Mike Hommey [:glandium]]

Comment on attachment 752361 [details] [diff] [review]
add MozbuildObject.from_environment

Review of attachment 752361 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/base.py
@@ +131,5 @@
> +                break
> +
> +        if not topsrcdir:
> +            # See if we're running from a Python virtualenv that's inside an objdir.
> +            mozinfo_path = os.path.join(os.path.dirname(sys.prefix), "mozinfo.json")

does sys.prefix return $objdir/_virtualenv on windows too?

@@ +153,5 @@
> +        # another objdir, we abort. The reasoning here is that if you are
> +        # inside an objdir you probably want to perform actions on that objdir,
> +        # not another one.
> +        if topobjdir and config['topobjdir'] \
> +            and os.path.normpath(topobjdir) != os.path.normpath(config['topobjdir']):

might be better to use os.path.samefile on non-Windows.

Comment 13

[(not currently active) Ted Mielczarek]

I don't have an actual Windows objdir handy, but local testing in a VM indicates that sys.prefix gets set appropriately:
$ python /z/build/mozilla-central/python/virtualenv/virtualenv.py  ./ve
New python executable in ./ve\Scripts\python.exe
Installing setuptools................done.
Installing pip...................done.

$ ./ve/Scripts/python.exe
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.prefix
'c:\\Documents and Settings\\mozillabuilder\\ve'

Comment 14

[(not currently active) Ted Mielczarek]

Pushed this to try to sanity check that it doesn't break anything:
https://tbpl.mozilla.org/?tree=Try&rev=f3e855054b2c

Comment 15

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8a990626f354

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/8a990626f354