782958 - Add pip and virtualenv to MozillaBuild

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, task)

Description

[Ed Morley [:emorley]]

It would be useful to add pip to MozillaBuild, so we don't have to easy_install it whenever working on projects that use it for setup (eg OrangeFactor, bug 782953).

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Not going to commit to it at this point, but I'll look into this.

Comment 2

[Gregory Szorc [:gps]]

Add virtualenv and you get pip for free. FWIW, we already have virtualenv in m-c. If you create a virtualenv via m-c, that virtualenv will have pip in it. That should work in MozillaBuild today.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Comes for free with Python 2.7.9.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

I have this mostly-working, except for one ugly issue.

As alluded to by http://hg.mozilla.org/mozilla-build/file/58edf3344962/packageit.py#l160, the path to the python executable gets hard-coded during the installation process. In the current world, it gets baked into python/Scripts/*.exe (pip.exe, virtualenv.exe, etc). Looking at the page referenced in that comment, I guess that means I need to somehow make use of the --executable flag to override that, but I'm not exactly sure how that works yet.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

So IIUC, basically munge_easy_install_script is going to have to be rewritten to find the <stagedir>\mozilla-build\python\python.exe line in each exe and re-write it to just python.exe without corrupting the end result. Fun.

Comment 6

[(not currently active) Ted Mielczarek]

It's been so long since I wrote that code that I don't remember anything about it. Rewriting exes sounds horrible, but if it's just C strings inside the binary that you're rewriting to a shorter string it should be doable.

Alternately you could hack away at the stuff that's causing full paths to get inserted there, but that might be a losing battle.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

I manually modified the path with Notepad++ and it appears to have worked ok, so I think that's going to be the way to go when I get back from vacation.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

I shamelessly cribbed some example code off StackOverflow that appears to be working nicely.

As it stands currently, I'm just downloading and installing whatever the latest version available is from pip (vs. installing an in-tree version or specifying the version when downloading from pip). Does anybody object to that? I don't see any major issues just going with the latest. And if we find some future version of one of the packages to be problematic, we can always fix it to a specific version then.

And finally, a test build (including hg 3.2.4 from bug 545432):
http://people.mozilla.org/~rvandermeulen/MozillaBuildSetup2.0.0pre.exe (78.1MB)
SHA1: 21887713c187744a72081ba9ac1574e37c894411

I definitely recommend not installing this over top of an existing install. Feedback welcome!

Comment 9

[Gregory Szorc [:gps]]

Installing latest version from pip at time of the MozillaBuild install is somewhat concerning. One of the properties of today's MozillaBuild that I like is it is deterministic. If you have a specific MozillaBuild installer/binary, you are guaranteed certain results. And this means that Firefox builds are more easily deterministic and reproducible. Downloading the latest version of pip, virtualenv from the web would change that.

I favor embedding the latest pip/virtualenv release in the MozillaBuild installer. Or, at least pin the version number in the installer.

Those concerns aside, using the latest versions of pip and virtualenv should have minimal risk beyond determinism concerns.

I'm very curious to see all the patches to MozillaBuild! Thanks so much for doing this work.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Pulling the latest versions from pip occurs during installer packaging. Nothing is pulled from the web during the installation process :)

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Attachment: patch

This patch:
1) Runs ensurepip from python 2.7.9, then updates pip & setuptools to their latest versions (not sure why ensurepip doesn't install the latest to begin with).
2) Installs virtualenv from pip.
3) Removes the hardcoded paths to python.exe from each shebang in the python Scripts directory.
4) Removes obsolete components from the MozillaBuild repo.

Note that all of the above happens during packaging, not as part of the client installation.

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8551428 [details] [diff] [review]
patch

Review of attachment 8551428 [details] [diff] [review]:
-----------------------------------------------------------------

What could possibly go wrong?

Comment 13

[Ryan VanderMeulen [:RyanVM]]

http://hg.mozilla.org/mozilla-build/rev/05664c670e65