891474 - Establish unified directory for Python build actions

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

We have a number of miscellaneous Python scripts in the tree that perform custom build actions. e.g. purge manifests, nsinstall, xpccheck.py. Many of these are in /build and /config.

I would like to unite all of the scripts under a common directory and provide a common mechanism for invoking them from make files.

Benefits:

* We encourage build actions to be written as importable Python modules, not as scripts. This fosters better/easier testing and reusability.

* If we put all scripts in a package on sys.path, this should eliminate some of the copies in /js/src.

* Easier to employ pymake magic to universally inline scripts (as opposed to running a new process/interpreter).

* Help cut down on pythonpath.py abuse.

Patch forthcoming.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Establish unified directory for Python build actions

I created python/mozbuild/mozbuild/action to hold scripts/modules that
can be used as build actions. I moved the purge manifests and xpcshell
manifest auditing scripts there.

I created a "py_action" callable in config.mk that simply invokes said
scripts from the well-defined location.

This seems to "just work." I stopped short of employing pymake % magic.

Seeking sign-off from multiple build peers because this idea may come
across as somewhat radical.

Comment 2

[Gregory Szorc [:gps]]

I was playing around with pymake % magic locally and doing it right will be a little bit of effort.

Currently, the sys.path of pymake doesn't contain the virtualenv. However, you can define $(PYCOMMANDPATH) to be a directory to append to sys.path. We currently use $(topsrcdir)/config in config.mk.

Unfortunately, it looks like pymake internally treats this variable as a string, so assigning multiple word-delimited paths to this variable does not work. Furthermore, since $(topsrcdir) resolves differently in /js/src, we won't be able to reference /python/mozbuild easily (or is there a backdoor somewhere - I can't recall).

I think what I'm trying to say is that I think we need to properly hook pymake up the virtualenv and that this is a prerequisite to native/% execution of build actions from pymake.

Comment 3

[(not currently active) Ted Mielczarek]

I agree with this. The Python native command stuff was implemented before we had a virtualenv, so it didn't really take that into account. PYCOMMANDPATH works with either space or os.sep-separated paths:
http://mxr.mozilla.org/mozilla-central/source/build/pymake/tests/native-pycommandpath-sep.mk
http://mxr.mozilla.org/mozilla-central/source/build/pymake/tests/native-pycommandpath.mk

...but ideally it'd just be using the virtualenv. That might be a PITA, I don't know.

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 772781 [details] [diff] [review]
Establish unified directory for Python build actions

(splinter annoyingly hides changes with no diff)

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 772781 [details] [diff] [review]
Establish unified directory for Python build actions

Review of attachment 772781 [details] [diff] [review]:
-----------------------------------------------------------------

Seems fine. I could bikeshed about the naming of the directory, but I don't care that much.

Comment 6

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3abc4c4a4f65

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3abc4c4a4f65

Comment 8

[Philip Chee]

comm-central bustage fix:
https://hg.mozilla.org/comm-central/rev/c5397c5f0faf