Closed Bug 545432 Opened 14 years ago Closed 9 years ago

Don't use py2exe based Mercurial

Categories

(Firefox Build System :: MozillaBuild, task)

Product:
Firefox Build System
Component:
MozillaBuild
Platform:
x86
Windows XP
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: whimboo, Assigned: RyanVM)

References

(
URL
)

Details

Attachments

(3 files, 6 obsolete files)

hg33-color.patch
6.94 KB, patch
Details | Diff | Splinter Review
Install hg from pypi instead of the MSI installer
11.97 KB, patch
gps
: review+
Details | Diff | Splinter Review
Import the hg33-color patch and user better defaults
9.06 KB, patch
gps
: review+
Details | Diff | Splinter Review 
Lately we had some troubles to use Mercurial from within Python on Windows. MozillaBuild is installing Mercurial under 'hg' but that particular version is not intended to be used from Python. Instead a version specific to the installed Python version has to be installed. An installer can be downloaded from the website below.

http://bitbucket.org/tortoisehg/thg-winbuild/downloads/

It would be helpful when we can add Mercurial as a default package.
Blocks: 537840
Users under Windows can't easily install it on their own without having a compiler from Visual Studio installed on the system. It would be really helpful to see it in the next version of MozillaBuild.
There are no more official builds available for Python 2.5. I have put the latest one (v1.4.3) on my people account:

http://people.mozilla.com/~hskupin/downloads/mercurial-1.4.3.win32-py2.5.exe
(In reply to comment #1)
> Users under Windows can't easily install it on their own without having a
> compiler from Visual Studio installed on the system. It would be really helpful
> to see it in the next version of MozillaBuild.

Henrik, Mozilla Build cannot install a Visual Studio compiler on the user's system.  We are not allowed by microsoft to rebundle that code and install it ourselves.  And we would seriously hork a developer's existing installations of visual studio if we tried it.

Why do you need this module in the first place?

Why can't you just call out to shell to excercise hg commands.  I'm not clear on what issue you're solving.
(In reply to comment #3)
> Henrik, Mozilla Build cannot install a Visual Studio compiler on the user's
> system.  We are not allowed by microsoft to rebundle that code and install it
> ourselves.  And we would seriously hork a developer's existing installations of
> visual studio if we tried it.

We wouldn't have to. The link I have provided is the latest version of the module and doesn't need any compilation.

> Why do you need this module in the first place?
> 
> Why can't you just call out to shell to excercise hg commands.  I'm not clear
> on what issue you're solving.

I can call the shell commands but having it included as module gives much more control when failures occur.

Ted, would you also tend to mark this bug as Wontfix? If yes, I would have to update my scripts to always use the system call to execute hg commands. I can live with that, but having a decision on it soon would be great. Thanks.
(In reply to comment #4)
> (In reply to comment #3) 
> We wouldn't have to. The link I have provided is the latest version of the
> module and doesn't need any compilation.
 
What happens if we put a pre-compiled binary onto the system and we upgrade our hg support to something greater than 1.4.3?  Would we then have to build this component too?  I'm not sure what your strategy is for future updates.

> > Why do you need this module in the first place?
> > 
> > Why can't you just call out to shell to excercise hg commands.  I'm not clear
> > on what issue you're solving.
> 
> I can call the shell commands but having it included as module gives much more
> control when failures occur.
What commands are you doing?  I think if you are always only doing one command then that is a much different scenario than if you're doing a bunch of different commands.
I don't have any problem with shipping this with MozillaBuild. MB ships with fixed versions of Python and Hg anyway.
(In reply to comment #5)
> What happens if we put a pre-compiled binary onto the system and we upgrade our
> hg support to something greater than 1.4.3?  Would we then have to build this
> component too?  I'm not sure what your strategy is for future updates.

Both will be independent from each other. And they shouldn't cause conflicts.

> What commands are you doing?  I think if you are always only doing one command
> then that is a much different scenario than if you're doing a bunch of
> different commands.

I'm using it to clone or pull a repository, and update it to a specific branch.

But after more thinking about it, it would probably better to rely on the systems hg command and use a system call to execute that specific command. Shouldn't be that hard to modify my code. Also it turned out in bug 568990 that we can run into trouble with older versions of the hg module installed on a system.

Shall we close it as WONTFIX?
I'm weird in the way that I every now and then like to use mercurial straight in the python shell.

I'd favor having an hg in MozillaBuild that can just do that.
Ted, could we try to get this into MozillaBuild then? I even like to use the libs directly as having to call system commands, which are involved in parsing the output.
I started looking into this, and got it mostly working (building Mercurial from source using the Python in MozillaBuild), but there are a couple of hg internals that fall apart if you're not running via hg.exe on Windows, and there are a couple of other things that don't work right if you don't have PyWin32 installed (which I failed to accomplish in bug 496535 originally).
I was able to integrate the mercurial package into my own stripped-down environment. Therefore I have used the pre-compiled binary from:

http://bitbucket.org/tortoisehg/thg-winbuild/downloads/

Ted, which hg internals fall apart?
URL: http://mercurial.selenic.com/bts/issu...
Ok, while working on getting Mercurial installed in our Mozmill environment, I have seen that you can install mercurial manually just by doing the following steps:

1. Download the latest gz file from pypi
2. Install it by using the --pure option

Downside is that it will not install the compiled c-code files but only the slower Python implementation. For our use it is ok.

So it's up to you if we should include it or not. If you don't think it's worth lets close this bug as wontfix.
I was playing with this last night because of bug 728117.

I was unable to successfully compile HackableMercurial using the instructions at http://mercurial.selenic.com/wiki/HackableMercurial. I tried with Visual Studio 2008 and MinGW and I kept running into DLL import failures when I ran hg.exe. The binary extensions were built, but nothing except a pure Python / no binary extensions worked.

I was, however, able to download and install the source installer (http://mercurial.selenic.com/release/windows/mercurial-2.9.win32-py2.7.exe). It installed things into my c:\Python27\Lib\site-packages\ and c:\Python27\Scripts directory. That hg installation appears to be using the compiled extensions (things are pretty fast) and it seems to work when executed inside my MozillaBuild environment. Although there could be msys compatibility issues I'm not seeing yet.

I don't believe we can switch MozillaBuild to install from this installer because it compiles the extensions at installer time and this requires Visual Studio 2008 (I think). But, I think we can archive the results of this build and decompress it into MozillaBuild. We should be able to do that plus drop a .pth into mozilla-build\python\Lib\site-packages and all should be well. We may even be able to create an egg or wheel.

RyanVM: is this something you can look at for the next MozillaBuild release?
Flags: needinfo?(ryanvm)
Summary: Install Python module for Mercurial per default in MozillaBuild → Don't use py2exe based Mercurial
If by "next" you don't mean 1.9, sure.

FWIW, I'm also looking at MSYS2, which includes it's own build of Mercurial. Don't know how that would work in this situation.
Flags: needinfo?(ryanvm)
This shouldn't block 1.9.

The requirements for a solution are:

1) Mercurial should not ship alongside its own python27.dll
2) Mercurial binary extensions must work
I installed msys2 + mercurial and it seems to be packaged properly.

I look forward to our new msys2 based MozillaBuild.
So we don't want to continue with a possible replacement of msys by conemu (bug 505582)?
(In reply to Henrik Skupin (:whimboo) from comment #18)
> So we don't want to continue with a possible replacement of msys by conemu
> (bug 505582)?

Conemu (which is great - I love it) is a console emulator rather than a replacement for msys, so this doesn't affect bug 505582.
Oh, sorry. you are right. I totally mixed up mintty with msys.
Blocks: 1023129
I have hg installing via pip mostly working now (including compiled code) thanks to the machinery I'm getting created for bug 782958 (dropped the installer size by ~3MB too!). One question is what should be done about the Mercurial.rc patching we normally do in packageit.sh (http://hg.mozilla.org/mozilla-build/file/58edf3344962/packageit.sh#l17). The patch in question - http://hg.mozilla.org/mozilla-build/file/58edf3344962/Mercurial.rc.patch.

I'm not sure where to make these changes within the python-based directory structure instead.

Also, how should I go about benchmarking this vs. the py2exe distribution we've been using to date?
Flags: needinfo?(ted)
Flags: needinfo?(gps)
Assignee: nobody → ryanvm
Blocks: MozillaBuild2.0
(In reply to [Away 24-Dec to 4-Jan] Ryan VanderMeulen [:RyanVM UTC-5] from comment #21)
> I have hg installing via pip mostly working now (including compiled code)
> thanks to the machinery I'm getting created for bug 782958 (dropped the
> installer size by ~3MB too!). One question is what should be done about the
> Mercurial.rc patching we normally do in packageit.sh
> (http://hg.mozilla.org/mozilla-build/file/58edf3344962/packageit.sh#l17).
> The patch in question -
> http://hg.mozilla.org/mozilla-build/file/58edf3344962/Mercurial.rc.patch.

I'd say those changes should go in the user specific hgrc not the global config. As such, either MozillaBuild could check if there's an hgrc, and if not create one in ~/ or else leave this to |mach mercurial-setup|
Also need to figure out what if anything needs doing for this:
http://hg.mozilla.org/mozilla-build/file/58edf3344962/installit.nsi#l51
I was just able to clone a repo using https without issue, though. So maybe we can just drop it.
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #24)
> I was just able to clone a repo using https without issue, though. So maybe
> we can just drop it.

Note that mach mercurial-setup sticks [hostfingerprints] in ~/.hgrc, so that might confuse the issue. Try again without that? Also, on my Linux system there's /etc/mercurial/hgrc.d/cacerts.rc, so presumably there's an equivalent directory in the MSYS install in which we can dump that file.

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #21)
> I have hg installing via pip mostly working now (including compiled code)
> thanks to the machinery I'm getting created for bug 782958 (dropped the
> installer size by ~3MB too!). One question is what should be done about the
> Mercurial.rc patching we normally do in packageit.sh
> (http://hg.mozilla.org/mozilla-build/file/58edf3344962/packageit.sh#l17).
> The patch in question -
> http://hg.mozilla.org/mozilla-build/file/58edf3344962/Mercurial.rc.patch.
> 
> I'm not sure where to make these changes within the python-based directory
> structure instead.

Do we not wind up with an equivalent Mercurial.rc file anywhere in the Python install? I'd be fine with punting on this and just making sure that `mach mercurial-setup` tweaks all these knobs on Windows. The only things I think are really important there are the color/pager setup and that's not really going to break anything (just not look as nice).
Flags: needinfo?(ted)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #26)
> Note that mach mercurial-setup sticks [hostfingerprints] in ~/.hgrc, so that
> might confuse the issue. Try again without that? Also, on my Linux system
> there's /etc/mercurial/hgrc.d/cacerts.rc, so presumably there's an
> equivalent directory in the MSYS install in which we can dump that file.

Ah, good call. With the [hostfingerprints] section commented out from my ~/.hgrc:

warning: hg.mozilla.org certificate with fingerprint af:27:b9:34:47:4e:e5:98:01:f6:83:2b:51:c9:aa:d8:df:fb:1a:27 not verified (check hostfingerprints or web.cacerts config setting)

I'll see what I can figure out, though I'm wondering if it's even needed given that mach mercurial-setup takes care of it otherwise.

> Do we not wind up with an equivalent Mercurial.rc file anywhere in the
> Python install? I'd be fine with punting on this and just making sure that
> `mach mercurial-setup` tweaks all these knobs on Windows. The only things I
> think are really important there are the color/pager setup and that's not
> really going to break anything (just not look as nice).

I don't see how to set any global options via this install method.
Though perhaps putting it inside MSYS' /etc/mercurial/hgrc would work.
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #27)
> I'll see what I can figure out, though I'm wondering if it's even needed
> given that mach mercurial-setup takes care of it otherwise.

You can't run mach mercurial-setup without having cloned m-c, so that's a bit chicken-and-egg...

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #28)
> Though perhaps putting it inside MSYS' /etc/mercurial/hgrc would work.

That seems like the right place.
This is what I was talking about earlier:
http://selenic.com/repo/hg-stable/file/4308087f2fbd/mercurial/scmwindows.py

This is what hg on Windows does to find mercurial.ini or hgrc.d. That util.executablepath() call there will return the path to python.exe in your new setup, so worst case we can put a mercurial.ini or hgrc.d next to the python binary.

>>> import mercurial.util
>>> mercurial.util.executablepath()
'c:\\mozilla-build2\\python\\python.exe'
The consulted paths for hgrc files is actually more complicated than what's been referenced: http://selenic.com/repo/hg/file/d944492445fa/mercurial/help/config.txt#l45

No shortage of options for overrides. I'd argue for changing the per-installation defaults for the MozillaBuild installation so e.g. they don't interfere with TortoiseHg.

For performance, having the .pyd compiled C extensions is a must. The other concern is startup speed. We don't want to regress `time hg --version` (too much). Testing startup overhead of an install with firefoxtree, bzpost, reviewboard, and a good selection of built-in selections would also be a good test. Pro tip: use |HGRCPATH=/dev/null hg| to wipe out the non-repo-local configs so you can easily test vanilla configs.

Oh, and `hg debuginstall` will tell you if the new installation is sane.
Flags: needinfo?(gps)
(In reply to Gregory Szorc [:gps] from comment #31)
> The consulted paths for hgrc files is actually more complicated than what's
> been referenced:
> http://selenic.com/repo/hg/file/d944492445fa/mercurial/help/config.txt#l45
> 
> No shortage of options for overrides. I'd argue for changing the
> per-installation defaults for the MozillaBuild installation so e.g. they
> don't interfere with TortoiseHg.

By that you mean dropping files in the MSYS /etc directory?

> For performance, having the .pyd compiled C extensions is a must.

Done :)

> The other concern is startup speed. We don't want to regress `time hg --version` (too
> much).

Ted and I were testing this earlier today. For both of us, there was a bit of a difference on the first run, but subsequent ones were pretty close (~100ms for me for both hg.exe and python-hg). Some sort of caching I guess. Anyway, didn't appear to be overly significant overall.

> Testing startup overhead of an install with firefoxtree, bzpost,
> reviewboard, and a good selection of built-in selections would also be a
> good test.

Ted also mentioned things like cloning a repo from another local clone. Does that sound like a good idea? I could also try common operations like update, merge, etc I'd think.

> Pro tip: use |HGRCPATH=/dev/null hg| to wipe out the
> non-repo-local configs so you can easily test vanilla configs.

Nice, thanks for the tip!

> Oh, and `hg debuginstall` will tell you if the new installation is sane.

$ hg debuginstall
checking encoding (cp1252)...
checking Python executable (c:\mozilla-build-test\python\python.exe)
checking Python version (2.7.9)
checking Python lib (c:\mozilla-build-test\python\lib)...
checking installed modules (c:\mozilla-build-test\python\Lib\site-packages\mercurial)...
checking templates (c:\mozilla-build-test\python\Lib\site-packages\mercurial\templates)...
checking commit editor...
checking username...
no problems detected
I started poking around with Windows Mercurial packaging as well. If you need some help here, I can justify spending time to work on it since version control is part of my job responsibilities :)
Attached patch checkpoint patch (obsolete) — — Splinter Review 
Not ready for review yet since this still requires sorting out setting the global options (I haven't had a chance to work on it, personally) and better out of the box color handling. Comment 23 needs addressing too, but I'm thinking for that it's probably easiest to just add the appropriate hostfingerprints section to whatever global options we create?

I haven't had a chance to do any benchmarking either yet, other that to say that I haven't noticed any obvious differences using it on a daily basis. I would welcome any and all help on that with the build linked from bug 782958 comment 8.

The main issue I have with this current setup is that the color extension doesn't play nicely w/ the Windows command prompt AFAICT. It generates gibberish characters at the beginning and end of each line rather than coloring them. This can be worked around by running the command through the pager extension, which then allows for proper coloring. However, that's kinda crappy IMO since there are at least some commands I'd prefer *not* being paged by default.

Though maybe the issue is that we're specifying ansi mode instead of win32?
http://mercurial.selenic.com/wiki/ColorExtension

Otherwise, the link below has some other suggestions for what to do:
http://asimilatorul.com/index.php/2010/05/16/mercurial-pager-with-color-support-on-windows/
Attachment #8551431 - Flags: feedback?(gps)
Just did an experiment of Python+hg vs. py2exe Hg (as shipped with MB 1.11). I did a local clone of mozilla-central to a fresh repo. I repeated it 3 times for both versions. The times were essentially identical for both and showed the same amount of consistency between runs. As previously mentioned, `time hg --version` shows similar results between the two versions as well.

Greg, I'd love for you to do some more detailed experiments with scenarios that interest you, but at least from the testing I've done so far, there's little reason to believe we're taking any serious performance penalties here (yay!).
Comment on attachment 8551431 [details] [diff] [review]
checkpoint patch

Review of attachment 8551431 [details] [diff] [review]:
-----------------------------------------------------------------

Looking good!

For color, the extension uses win32 on Windows unless TERM is set. TERM *is* set in the Mozilla-Build terminal, so color uses "ansi".

I don't think cloning is a good benchmark. Too much variability due to network conditions. Instead, try local commands, like `hg status` (also highly dependent on state of filesystem cache), `hg log -r tip`, and even `hg qpush` or `hg qpop`. As long as the execution time of these short-lived commands doesn't regress significantly, we should be fine. My concern is that loading individual .py files will be significantly slower than loading an indexed zip file (due to Windows I/O badness).

I'll try to find some time to test this.
Attachment #8551431 - Flags: feedback?(gps) → feedback+
Ryan: Where can I find an installer to test this?
Flags: needinfo?(ryanvm)
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #34)
> The main issue I have with this current setup is that the color extension
> doesn't play nicely w/ the Windows command prompt AFAICT. It generates
> gibberish characters at the beginning and end of each line rather than
> coloring them. This can be worked around by running the command through the
> pager extension, which then allows for proper coloring. However, that's
> kinda crappy IMO since there are at least some commands I'd prefer *not*
> being paged by default.

Note that this is the motivation behind a big chunk of the Mercurial.rc.patch:
http://hg.mozilla.org/mozilla-build/file/tip/Mercurial.rc.patch#l68

The color/pager bits there handle it pretty well out of the box.
mach mercurial-setup doesn't work :(

I haven't looked into this. My guess is anything that invokes `hg` will need to be taught to execute it within the context of an msys shell or via python.exe. I believe somewhere there is a very basic C program that produces a hg.exe which simply calls out to python.exe...

gps@gps-PC /c/dev/src/mozilla-central
$ ./mach mercurial-setup
I'm going to help you ensure your Mercurial is configured for optimal
development on Mozilla projects.

If your environment is missing some recommended settings, I'm going to prompt
you whether you want me to make changes: I won't change anything you might not
want me changing without your permission!

If your config is up-to-date, I'm just going to ensure all 3rd party extensions
are up to date and you won't have to do anything.

To begin, press the enter/return key.

Error running mach:

    ['mercurial-setup']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

WindowsError: [Error 193] %1 is not a valid Win32 application

  File "c:\dev\src\mozilla-central\tools/mercurial/mach_commands.py", line 42, i
n mercurial_bootstrap
    result = wizard.run(map(os.path.expanduser, config_paths))
  File "c:\dev\src\mozilla-central\tools/mercurial\hgsetup\wizard.py", line 205,
 in run
    hg_version = get_hg_version(hg)
  File "c:\dev\src\mozilla-central\python/mozversioncontrol\mozversioncontrol\__
init__.py", line 19, in get_hg_version
    info = subprocess.check_output([hg, 'version'], env=env)
  File "c:\dev\mozilla-build-beta\python\lib\subprocess.py", line 566, in check_
output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "c:\dev\mozilla-build-beta\python\lib\subprocess.py", line 710, in __init
__
    errread, errwrite)
  File "c:\dev\mozilla-build-beta\python\lib\subprocess.py", line 958, in _execu
te_child
    startupinfo)
Other than that issue, the performance of the new Mercurial package is fine!

Also, I'm able to `hg --debugger` and import and run json, xmlrpc, and all the other modules which we historically haven't been able to use. This is going to make so many problems go away. Did you know that we've had to proxy Review Board API calls through the Mercurial wire protocol as part of MozReview to work around the fact that Windows Mercurial clients don't have access to all the necessary Python modules? I can't wait to remove those hacks.
*sigh* I assumed that's what hg.bat in the Scripts directory was for :(
You are assuming the "locate hg program" logic is taking .bat files into account, which it may not :)
Greg, I can continue to work on the config issues, but I'm afraid that fixing up the build system and mach is beyond what I'm going to be able to do.

Right now, I think I'm going to target getting this shipped after Mercurial 3.3.1 is released in early March. Is that enough time that you'd be able to help get that work done?
Flags: needinfo?(gps)
If you push a new MozillaBuild out tomorrow, I'll have mach patched tomorrow :)

I don't think there is a need to wait until 3.3.1. If everything else is ready sooner, let's move forward with 3.2.4. Or, if a few weeks have passed and 3.3 is looking pretty stable, we may even take that.
Flags: needinfo?(gps)
Depends on: 1128586
(In reply to Gregory Szorc [:gps] from comment #31)
> The consulted paths for hgrc files is actually more complicated than what's
> been referenced:
> http://selenic.com/repo/hg/file/d944492445fa/mercurial/help/config.txt#l45

After a lot of trial and error (and playing with the firehose that is ProcMon), dropping mercurial.ini into the root python dir is what ended up working. Not exactly intuitive, but oh well :)
I'm still having issues with the color extension out of the box unless I change mode from ansi to win32 :\
To elaborate:

Out of the box settings (mode = ansi):
←[0;33mchangeset:   249:3c3b572ac5ac←[0m

Overriding with mode = win32, I get proper coloring.

$ echo $TERM
cygwin
This is ready for review now. Default configs are working as expected, though the color issue still needs to be resolved.
Attachment #8551431 - Attachment is obsolete: true
Attachment #8558659 - Flags: review?(gps)
I can reproduce the issues with the color extension on a stock Win7 VM as well. Unless I hear otherwise from someone, I'm going to use mode = win32 instead of ansi as that seems to work consistently.
I'm using color.mode=ansi with the default MozillaBuild terminal and color is working for me! If I manually run --config color.mode=win32, color actually disappears from the terminal.

Are you sure you are using the MingW terminal (which is based on cmd.exe) and not MinTTY or some such?
Flags: needinfo?(ryanvm)
Yep, this is with a terminal started directly from the start-msvc batch files, which launch bash.
Flags: needinfo?(ryanvm)
Fixes line endings in mercurial.ini. Should all be LF now.
Attachment #8558726 - Flags: review?(gps)
Attachment #8558659 - Attachment is obsolete: true
Attachment #8558659 - Flags: review?(gps)
This is the exact mercurial.ini I'm using in the clean Win7 VM (including the LF fix).
Attached image screenshot w/ ansi (obsolete) — 
Screenshot with mode = ansi set.
Attached image screenshot w/ win32 (obsolete) — 
Changing the mode to win32 and nothing else
Comment on attachment 8558659 [details] [diff] [review]
Install hg from pypi instead of the MSI installer

Review of attachment 8558659 [details] [diff] [review]:
-----------------------------------------------------------------

::: mercurial.ini
@@ +4,5 @@
> +mq =
> +pager =
> +progress =
> +purge =
> +rebase =

Let's add share as well.

@@ +5,5 @@
> +pager =
> +progress =
> +purge =
> +rebase =
> +transplant =

Let's turn on blackbox to get forensic logging from clients so when people report performance problems we can look at the logs to see what took so much time.

We have blackbox enabled on hg.mozilla.org. Facebook runs it on all their client machines. It should be globally enabled.

[extensions]
blackbox =

[blackbox]
track = *
maxsize = 10 MB
maxfiles = 2

@@ +8,5 @@
> +rebase =
> +transplant =
> +
> +[alias]
> +show = log --patch --verbose --rev

This is a silly alias. It is roughly equivalent to `hg export`

@@ +11,5 @@
> +[alias]
> +show = log --patch --verbose --rev
> +
> +[color]
> +mode = ansi

As I just commented, mode=win32 works for me. I'm running in a fresh Windows 7 VM.

@@ +21,5 @@
> +
> +[pager]
> +pager = less.exe -mnFRSXQ
> +quiet = True
> +attend = diff,status,log,qdiff,blame,annotate,pdiff,glog,show,qseries

Only put glog in this list if the glog alias is defined.
Attachment #8558659 - Attachment is obsolete: false
I've submitted 5 patches to Mercurial to make color behave better in MSYS.

We have a mismatch between the MSYS terminal and GNU less. The MSYS terminal is supporting win32 color codes. However our pager, GNU less, supports ANSI sequences. My patches teach Mercurial to recognize non-ANSI terminals on Windows better *and* they add a color.pagermode that can be used to define the color mode if Mercurial's pager is active. After the patches, we'll just need to set color.pagermode=ansi in MozillaBuild. We'll be able to remove the configs activating the pager on nearly every command since color will just work out of the box.

RyanVM: this will mean applying some patches on top of the 3.3 source release before we install Mercurial, as my patches won't make it until Mercurial 3.4. Sorry :/
Yeah, I suspected that was how it was going to end up. That's fine, though. The only real concern with that is end users hitting unexpected behavior changes if they update locally from pypi. However, unless they go through the trouble of setting up a Visual C++ for Python environment and all, that's likely not going to be a recommended practice anyway.

Do you mind attaching a patch roll-up to this bug that I can use in-tree? Thanks for doing all this!
I'll attach a roll-up patch once things have been accepted upstream. That should take <48 hours unless there is resistance to my patches (which wouldn't surprise me, quite frankly).
Attached patch hg33-color.patch — — Splinter Review 
The last of my upstream patches has been accepted into Mercurial. They will be part of Mercurial 3.4.

Here is a roll-up of all those patches into a diff file. I grafted my patches on top of the 3.3 release tag. So this should apply without fuzz using `patch -p0`.
I forgot to mention that the hgrc should now contain the following entry:

[color]
pagermode = ansi

There is no need to define "mode" any more, as this patch will detect it properly from msys terminals.

There is no need to define a pager.attend option because that hack is no longer necessary now that color is working. Good riddance.
Comment on attachment 8558726 [details] [diff] [review]
Install hg from pypi instead of the MSI installer

Review of attachment 8558726 [details] [diff] [review]:
-----------------------------------------------------------------

I might as well cancel this since it will need updated to incorporate the upstream color/pager patches I made.
Attachment #8558726 - Flags: review?(gps)
Given the current state of my patch queue, it's actually a lot less messy to do the color changes in a new patch. So here's a rebased version of the main patch for review.
Attachment #8558659 - Attachment is obsolete: true
Attachment #8558726 - Attachment is obsolete: true
Attachment #8558727 - Attachment is obsolete: true
Attachment #8558728 - Attachment is obsolete: true
Attachment #8558730 - Attachment is obsolete: true
Attachment #8560938 - Flags: review?(gps)
Self-explanatory. I've confirmed that colors all behave as expected both for stand-alone commands and ones I force through pager \m/.

My only question is that if we remove the pager.attend options, that means we're basically not running anything through pager anymore? Are we sure that's what we want to do?
Attachment #8560939 - Flags: review?(gps)
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #66)
> My only question is that if we remove the pager.attend options, that means
> we're basically not running anything through pager anymore? Are we sure
> that's what we want to do?

*beyond the extensions defaults* that is :). And yeah, the more I think about it, the more I like that being something users can configure for themselves rather than us doing it globally for them.
Comment on attachment 8560938 [details] [diff] [review]
Install hg from pypi instead of the MSI installer

Review of attachment 8560938 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM.
Attachment #8560938 - Flags: review?(gps) → review+
Comment on attachment 8560939 [details] [diff] [review]
Import the hg33-color patch and user better defaults

Review of attachment 8560939 [details] [diff] [review]:
-----------------------------------------------------------------

::: packageit.sh
@@ +117,5 @@
> +# Patch Mercurial 3.3.x to better support color when running in MSYS
> +# This patch will be included in Mercurial 3.4+
> +pushd "${MSYS_STAGEDIR}/mozilla-build/python/Lib/site-packages"
> +patch -p1 -i "${MSYS_SRCDIR}/hg33-color.patch"
> +popd

This makes me a bit nervous. It /should/ work. But, the installation of Mercurial via pip will pre-generate .pyc files. I'd feel much better if we downloaded the Mercurial source archive, patched the source files, then ran `pip install .`

I'll approve this because it /should/ work. But it's not ideally how you should be doing Python packaging.
Attachment #8560939 - Flags: review?(gps) → review+
I hear what you're saying about that. I have verified that what's posted here does work as expected. Given the current packaging situation, it gets a bit tricky to move things around for what's basically a short-term fix until 3.4 is released, so I'm inclined to leave things as-is.

Would it make you feel better if we nuked those specific .pyc files as well before the final packaging to ensure that users don't end up on stale versions?
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #66)
> My only question is that if we remove the pager.attend options, that means
> we're basically not running anything through pager anymore? Are we sure
> that's what we want to do?

I agree with your later comment that configuring pager.attend beyond the defaults should be something users do.

I agree the proper packaging isn't worth the time given the intended short-term nature of the hack.

Instead of deleting .pyc files, how about running a Mercurial command before packaging? e.g. `hg identity https://hg.mozilla.org/mozilla-central` All changed files should get loaded by any `hg` invocation and cause .pyc files to get regenerated.
(In reply to Gregory Szorc [:gps] from comment #71)
> Instead of deleting .pyc files, how about running a Mercurial command before
> packaging? e.g. `hg identity https://hg.mozilla.org/mozilla-central` All
> changed files should get loaded by any `hg` invocation and cause .pyc files
> to get regenerated.

Works for me, and I've verified that it regenerates the pyc files as expected. :)

http://hg.mozilla.org/mozilla-build/rev/52ac048d4e65
http://hg.mozilla.org/mozilla-build/rev/22ae7d6a15ee
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Depends on: 1131048
Blocks: 906200
Product: mozilla.org → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: