728117 - Can't run bzexport from the Mercurial that ships with MozillaBuild

Status: RESOLVED DUPLICATE of bug 545432

(Firefox Build System :: MozillaBuild, task)

Description

[Brian Birtles (:birtles, Feb 26~ parental leave)]

bzexport[1] doesn't work for folks on Windows. Instead, we get:

  Error sending patch: No module named json

I believe it's because of the way Mercurial for Windows is packaged. Python is packaged with Mercurial using py2exe and py2exe apparently removes all unneeded modules, including 'json' in this case (and who knows, maybe others needed for bzexport).

One possibility is to use the Python version of Mercurial (maybe this is what bug 545432 is about?). The problem with that is it's incredibly slow. It is NOT a suitable replacement unless we can make it faster.

I currently have the Python version installed as well and aliased to 'hg.py' so I can run

  hg.py bzexport ....

And use the regular Mercurial binary other times.

I'm not sure how to solve this. Re-package Mercurial somehow? Rewrite bzexport to do json parsing/serialising by hand?

I think this deserves attention though because:

* We're encouraging people to split up their patches into more fine-grained pieces[2]
  As a result, bzexport is becoming an increasingly critical part of the development process (as if fixing bitrot on a 25 part patch queue wasn't bad enough, it often takes even longer still to re-add all the patches to Bugzilla!)
* We need more people developing on Windows

[1] http://hg.mozilla.org/users/tmielczarek_mozilla.com/bzexport/
[2] http://groups.google.com/group/mozilla.dev.planning/browse_thread/thread/58b7f166f22f44af/39eb2f69c03b21e9

Comment 1

[(not currently active) Ted Mielczarek]

Yeah, this sucks. Even if you get past that you'll hit other problems. The main issue here is that Hg on Windows is shipped as a py2exe-generated binary, which ships a stripped-down Python standard library. I investigated building our own Hg against the Python we have installed in bug 545432, but it's very fiddly and the resulting build is missing some things.

Comment 2

[David Teller [:Yoric] - still alive but not very active]

This would really make my life much easier. As it is, exporting a patch from my Windows VM is kind of nightmarish.

Comment 3

[Dirkjan Ochtman (:djc)]

Perhaps including hackable Mercurial would be helpful (I think MB already includes a Python, so perhaps you could change things a bit to re-use the Python stuff).

http://mercurial.selenic.com/wiki/HackableMercurial

Comment 4

[(not currently active) Ted Mielczarek]

Ooh, that sounds very much like what we'd want.

Comment 5

[Justin Wood (:Callek)]

:ted, who maintains MozillaBuild these days, can we get any traction on this.

Alternatively is bzexport able to make use of xmlrpc bmo libraries since the library.zip here has that in it.

Lastly until this is fixed, can you/someone add details about said issue (broken on windows... see Bug <>) to its readme https://hg.mozilla.org/hgcustom/version-control-tools/file/781bd278967d/hgext/bzexport/README

Comment 6

[Gregory Szorc [:gps]]

RyanVM maintains MozillaBuild these days.

I didn't realize we were using py2exe to distribute Mercurial with MozillaBuild.

The likely reason the source version is so slow is that it's not compiling/using the C extensions.

I'd like to think we could distribute Mercurial such that it uses the MozillaBuild python. Will definitely require some custom packaging magic. Easiest solution is likely to create an egg or to put a .pth in python/lib/site-packages. We could also hack up the .bat scripts to munge some environment variables. I'm not sure what all the trade-offs are. This seems doable to me.

Comment 7

[(not currently active) Ted Mielczarek]

(In reply to Gregory Szorc [:gps] from comment #6)
> I didn't realize we were using py2exe to distribute Mercurial with
> MozillaBuild.

We're not, we're using the official installers which are built that way.

> The likely reason the source version is so slow is that it's not
> compiling/using the C extensions.
> 
> I'd like to think we could distribute Mercurial such that it uses the
> MozillaBuild python. Will definitely require some custom packaging magic.
> Easiest solution is likely to create an egg or to put a .pth in
> python/lib/site-packages. We could also hack up the .bat scripts to munge
> some environment variables. I'm not sure what all the trade-offs are. This
> seems doable to me.

I tried building from source in bug 545432, I don't recall the specific issues but it was a pain.

I have no plans to work on this, sorry.

Comment 8

[Gregory Szorc [:gps]]

I also tried to build in a fresh msys environment and ran into issues. I think I've seen traffic on Mercurial's mailing lists about this not working. This could be "fun."

Anyway, I'm convinced this is a dupe of bug 545432 since the resolution is changing how Mercurial is packaged/shipped in MozillaBuild.