Deploy hg on windows as a regular python package

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps: Puppet
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: catlee, Assigned: markco)

Tracking

Details

(Whiteboard: [windows])

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
The current hg deploy on windows is a py2exe self-contained binary. This makes it hard to deploy other extensions for hg that rely on non-standard python libs.

Can we deploy hg as a regular python package?
If we do so, let's do the same on POSIX using the same Puppet code.

I wonder if the 'pip' provider for the 'package' type could do this on Windows?
I think doing this with puppet is the biggest win.
Component: RelOps → RelOps: Puppet
QA Contact: arich → dustin
(Reporter)

Comment 3

3 years ago
+1
Whiteboard: [windows]
Assignee: relops → mcornmesser
(Assignee)

Comment 4

3 years ago
It doesn't look like the pip provider will work on Windows:
Error: Could not set 'present' on ensure: Could not locate the pip command. at 6
5:/etc/puppet/environments/mcornmesser/modules/packages/manifests/mozilla/py27_m
ercurial.pp
Error: Could not set 'present' on ensure: Could not locate the pip command. at 6
5:/etc/puppet/environments/mcornmesser/modules/packages/manifests/mozilla/py27_m
ercurial.pp

But an easy_install will work. 

Using this method does not produce an hg.exe, so I am concerned with any hard coding looking for that exe file.
(Assignee)

Comment 5

3 years ago
Without the hg.exe I am pretty confident that there will be a failures when hg is being called. 

catlee: Is there someone I can bring this up to now, so we can stay ahead and move forward towards functioning?
Flags: needinfo?(catlee)
(Assignee)

Updated

3 years ago
Blocks: 1181597
(Assignee)

Comment 6

3 years ago
Created attachment 8633089 [details] [diff] [review]
BUG1144474.patch

This patch includes the deployment of HG bundleclone extension for Bug 1181597.
Attachment #8633089 - Flags: review?(jwatkins)

Comment 7

3 years ago
I see some references to MozillaBuild in this patch. If that's the MozillaBuild that developers use, version 2.0 contains a Mercurial installed as a package, leveraging the Python distribution inside MozillaBuild (rather than the py2exe foo).

Also, MozillaBuild is producing an hg.exe. There's a somewhat hidden build rule in Mercurial to produce the stub hg.exe binary. RyanVM can tell you more. Of course, using pip or easy_install should "just work." But hg.bat or hg.exe need to be in place so invocations via CreateProcess() work (CreateProcess() doesn't honor the shebang, unlike invocations within a UNIX-like shell process).
(Assignee)

Comment 8

3 years ago
We are unable to use MozillaBuild 2.0, https://bugzilla.mozilla.org/show_bug.cgi?id=1176111, so we are relying on an older version as a start point for the packages needed. This is only a stopgap until we have Windows package management in place. 

But is sounds like we maybe OK. There is an hg.bat that is generated by the easy_install.
Flags: needinfo?(catlee)
Keep in mind that unless you also install the Microsoft Visual C++ Compiler for Python 2.7 package from Microsoft [1], you'll incur a performance penalty when installing Mercurial as a python package as you'll need to also specify the --pure install option. Using --pure means falling back to python versions of performance-critical libraries that are otherwise compiled into native code.

And yes, I have experience also compiling the stub hg.exe. In our testing with MozillaBuild 2.0, we found that hg.bat didn't work very reliably by itself.

[1] http://www.microsoft.com/en-us/download/details.aspx?id=44266
(Assignee)

Updated

3 years ago
Attachment #8633089 - Flags: review?(jwatkins)
(Assignee)

Comment 10

3 years ago
catlee: After comment 9, with the need to add an additional compiler and to custom compile hg.exe, do we still want to move forward with this?
Flags: needinfo?(catlee)
(In reply to Chris AtLee [:catlee] from comment #0)

A bit more explanation/clarification on why this is an issue and why this bug was opened in the first place... Right now the hg package that we use has it's own, stripped-down, bundled copy of python instead of using the system python. 

When we want to install extensions, they may require other python modules that aren't part of the stripped-down one that's in the hg package. We hit this with the purge extension, for example. The fix that was needed for that extension required pywin32 bindings which existed in the system python, but not the hg python. TO force a fix, we would have to modify the version of python in the hg package and respin the whole thing again.

The proposed solution was to separate hg and have it use the system python so we didn't have issues like this. It wasn't that we didn't like having hg as an EXE (vs pip install), but that we wanted the segregation within the EXE of python and hg.
(Assignee)

Updated

3 years ago
Blocks: 1183547
No longer blocks: 1183542
Assignee: mcornmesser → arich
Flags: needinfo?(catlee)
Only meant to cancel the needinfo, not take the bug.

We're not going to implement this yet, since we believe we have a working exe install for windows and getting to AWS is more important than adding this extra functionality/doing this redesign.
Assignee: arich → mcornmesser
(Assignee)

Comment 13

2 years ago
Closing this bug. During the testing and building of 2008 configuration for AWS, hg as python package had a very poor performance.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
This was the --pure version I presume?
(Assignee)

Comment 15

2 years ago
Yes. Among many different configuration of installing it.
OK, I think that'd be expected then. It's why MozillaBuild goes the route of comment 9 :)
Under no circumstances should hg be installed without Python C extensions: the performance will be abysmal for mozilla-central.

Do you want Mercurial to publish wheels for Windows or will you go another route?

(I can make wheels happen if you need them.)
Flags: needinfo?(mcornmesser)
(Assignee)

Comment 18

2 years ago
 For the install I started here and tweaked and tried different combinations of versions of python and mercurial: https://www.mercurial-scm.org/wiki/WindowsInstall.

Wheels for Windows may be worth looking into, but that will need to a be a larger conversation.
Flags: needinfo?(mcornmesser)
I submitted upstream patches to Mercurial to produce wheels. Windows wheels should hopefully be uploaded to PyPI as part of the Mercurial 3.7 release in a few months. If you want, I can provide a wheel for 3.6 that you can experiment with. Just needinfo me.

Updated

2 years ago
Blocks: 1232699
You need to log in before you can comment on or make changes to this bug.