Closed Bug 505582 Opened 16 years ago Closed 11 years ago

Ship mintty/Console2/ConEmu-Maximus5 as the default MozillaBuild terminal

Categories

(Firefox Build System :: MozillaBuild, task)

Product:
Firefox Build System
Component:
MozillaBuild
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: robarnold, Assigned: RyanVM)

References

(
URL
)

Details

Attachments

(1 file, 4 obsolete files)

Add winpty to the package and use mintty by default
715.46 KB, patch
Details | Diff | Splinter Review
We've all had enough of the windows terminal. It's time for change.
Interesting idea...
I think it is sad our default developer experience is so crappy on Windows. We're not moving away from the terminal for building any time soon and the Cygwin terminal is junk. We can make the experience better. What needs to happen to make this feature request a reality? I'll also throw in mintty as an alternative. https://code.google.com/p/mintty/. You literally throw a .exe inside the msys/bin directory and it just works. Console2 *might* be more involved (I haven't used it). Either way, we probably rewrite the batch scripts to launch a different binary. Would there be a licensing concern with either of these? Console2 is GPLv2. mintty is GPLv3. Both are "just" terminals, so I don't think there is an issue of contamination. But, I'm no lawyer. Gerv?
After giving Console2 a test drive, my personal preference is towards mintty. The default UX is cleaner (no menus, tabs, or status bar). Although, Console2 does have some interesting features, including tabs and a few more configuration options. Also because of its non-minimal look, Console2 may be more friendly to Windows users not accustomed to terminals. I think either are vast improvements over what we have today.
Help me get the full picture: you want to ship a standalone <something>.exe, for which the source code is under GPL2 or GPL3, as part of the MozillaBuild package on Windows? That would be fine, but we would probably want to ship the source as part of the package, as this is much easier than other source distribution options the GPL offers. Gerv
I think the hard part here was doing the testing - Ted knows more about that since I think he did the last MozillaBuild release.
Attached patch Use mintty for terminal (obsolete) — — Splinter Review
Not sure how the packaging voodoo works. I have no way to test this. On my Windows machine, I dropped mintty.exe into mozilla-build\msys\bin\mintty.exe (it needs to be in the same directory as msys-1.0.dll). I verified the script changes to the start-* files worked as expected (although I only verified with the msvc-10 one). The switch to START is so the old terminal window closes after the new mintty window opens. We didn't need this before because we were launching bash, which installed itself into the currently-running terminal. mintty is a completely new terminal. Comparing the environments of terminals, the only difference is TERM. It switches from "cygwin" to "xterm."
Assignee: nobody → gps
Status: NEW → ASSIGNED
Attachment #635810 - Flags: review?(khuey)
We can switch back to Console2 if someone puts code forward :)
Summary: Ship Console2 as the default MozillaBuild terminal → Ship mintty as the default MozillaBuild terminal
(In reply to Gervase Markham [:gerv] from comment #4) > Help me get the full picture: you want to ship a standalone <something>.exe, > for which the source code is under GPL2 or GPL3, as part of the MozillaBuild > package on Windows? Correct. We would be shipping the pre-compiled binary. > That would be fine, but we would probably want to ship the source as part of > the package, as this is much easier than other source distribution options > the GPL offers. What do we actually need to do here? If we can just include a link somewhere to the original source, that would be preferred. Fewer moving parts, etc. I can attach a new patch with source code. Please spell out exactly what we need to do.
OK, I rechecked GPLv3 specifically. It's section 6d: http://www.gnu.org/copyleft/gpl.html We can just include a link, but that makes us responsible for checking to be sure the original server hasn't gone down. Better might be to stick a copy of the source tarball next to the MozillaBuild zip file on the FTP site, or wherever it lives, and note its location in the MozillaBuild README. As long as we don't update the binary, we never have to update the source. Does that make sense? Gerv
OK, that kinda makes sense. But, there is an additional wrinkle here I just thought of. MozillaBuild exists as both a "source" repository (https://hg.mozilla.org/mozilla-build/) (which contains pre-built binaries) and a single packaged binary (which users download). The thing that users download (i.e. is shipped) is essentially a glorified self-installing archive. Would it be sufficient to include the mintty source in the MozillaBuild source repository? The README file from the shipped archive would then point back to the hg.mozilla.org repository.
Attached patch Use mintty for terminal, v2 (obsolete) — — Splinter Review
Source archive is present in repository, but isn't shipped. Added README file to shipped archive saying where the HG repository is.
Attachment #635810 - Attachment is obsolete: true
Attachment #635810 - Flags: review?(khuey)
Attachment #635847 - Flags: review?(khuey)
Attachment #635847 - Flags: feedback?(gerv)
Comment on attachment 635847 [details] [diff] [review] Use mintty for terminal, v2 To me, this plan good it seems. Gerv
Attachment #635847 - Flags: feedback?(gerv) → feedback+
Was telling John Ford about this and he alerted me to an issue with mintty. Sure enough, many interactive programs like vim and even python (without -i) have issues. https://code.google.com/p/mintty/issues/detail?id=56 has all the gory details. This change may not be as trivial as I wanted :( Of course, we could still ship mintty as part of mozilla-build and encourage people to use it for building (which works just fine). It's just a few misc tools that are broken.
FWIW, Console2 doesn't have this issue (reasons explained in the Google Code issue). People complain about Console2's lag, etc. But, it *does* work and it is certainly better than the native Windows terminal. So, we may consider shipping Console2. Alternatively, we can investigate hooking https://github.com/rprichard/winpty up to mintty. Either way, more work for this bug.
vim and python are not "misc tools", unfortunately. I don't think we should ship a console in which those are broken. FWIW, I've been using Console2 for years now without any major problems; I'd suggest just shipping it. Alternatively, the latest versions of cygwin seem to come with a much nicer/newer console -- they don't just use cmd.exe any more. Might be worth investigating.
Oh hey, that new thing is mintty; ignore that :)
Last I recall Console2 had a lot of issues that made us not want to ship it as the default. The broken vim/python issue is probably a non-starter for us, unless there's a way we can work around this. (Ship a custom-built vim that can deal with mintty?) I think unfortunately we're going to find enough things are broken that makes it really frustrating.
whether or not they become the defaults, shipping Console2 and mintty with configurations that work with mozilla-build out of the box would be very nice. I've had to spend time each time i set up mozilla-build getting console2 or mintty going. Doing that inside of mozilla-build means that we don't have to spend time configuring them.
Ted: I agree we shouldn't ship with basic tools like vim and (interactive) python shell not working in the default terminal window. I agree with John that shipping mintty and/or Console2 would be very nice to have, even if they aren't available in the default terminal window. If we strip the .bat changes from the patch, we essentially have this. I'll continue to look at alternatives (including winpty) so we can get a better terminal by default without breakage. I'm pretty sure cygwin gets around the issue by ensuring tools like vim don't use the Win32 APIs for terminal interaction. Translated to mozilla-build, that would mean shipping more cygwin-like binaries instead of native Win32 ones. This is a rat hole we probably want to avoid.
This just includes the packaging changes. We can ship mintty for people to use if they want to. I know I will! We can leave the bug open to figure out integration (I still need to look at winpty).
Attachment #635847 - Attachment is obsolete: true
Attachment #638007 - Flags: review?(khuey)
FWIW, I use console2 from quite some time without any issue and I know many others do in Mozilla (as vlad pointed out above). Regarding your question whether Windows people would prefer to get a console with a menubar, sidebar and such, I personally disabled any additional ui but the console, they are useless and distracting. After all, the default command prompt has none of those.
I saw avaih drop a link to https://code.google.com/p/conemu-maximus5/ in IRC. Let's add it to the fray.
Summary: Ship mintty as the default MozillaBuild terminal → Ship mintty/Console2/ConEmu-Maximus5 as the default MozillaBuild terminal
http://mobaxterm.mobatek.net/ also seems quite useful, though quite more than a simple terminal emulator. It's got tabbed terminals, has a built in x-server if you need one, supports putty sessions, and works nicely with mozilla-build. It also has portable mode, so installation is not mandatory. Cool tool.
Comment on attachment 638007 [details] [diff] [review] Part 1: Add mintty to mozilla-build 11 months later ... let's be honest I'm not going to review this.
Attachment #638007 - Flags: review?(khuey)
Any reason to not use console2? It may have problems, but seemingly less than mintty? It can also use a xml file in the same directory as the config, so we could easily ship a config that has all the build envs set up as preset tabs as well. I guess no reason to not ship both even, they're tiny.
I'm fine with shipping whatever in the package. Making something the default is a higher bar, in that it can't break the interactive modes of frequently used programs. If something has solved the latter then let's go for it.
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #26) > ... in that it can't break the interactive modes of frequently used programs. Could you please define this problem more explicitly? I could test the conemu-maximus5 one from comment 22. I've been using it very happily for few months now.
I guess this was only mentioned offhand elsewhere in the bug, but the failure mode is that running things like `python` or `vim` without any arguments does not give you an interactive shell, since they can't detect that they're being run in a console.
With conemu, |python| gets me into the interactive shell, and |vim|... works normally (though I'm not a vim guy, so I might be missing something, but as far as I can tell, vim works well).
Interactive mode wfm with conemu too
That's probably good enough, thanks!
I just had a quick play with conemu -- looks good to me. I'll probably look at switching to it from Console2, since Console2's development has largely been abandoned. conemu can also store settings in an xml file, so we can just ship something nicely configured for launching the various startup scripts we have (vc10-x86, vc10-x64, vc11-{x86,64}, l10n, etc.).
I invited ConEmu's author to this thread to share his opinion and recommendations (version to use, etc). Hope he shows up.
At the moment, I think, stable version is preferred for build system. Any user can easily update any ConEmu installation with autoupdate feature. Well, there are some bugs in stable, which were are already fixed in latest versions, but it's hard to remember what exactly and how critical they was are... At last, I think, build system have no need in ANSI sequences processing?
(In reply to Maximus from comment #34) > At last, I think, build system have no need in ANSI sequences processing? Thanks for the info. If by ANSI sequences you mean colors, then it's quite handy. The color support is currently limited on windows, and with ConEmu I was to get more colors (during build, for hg, diff, etc, after some configuration). The console is used by Mozilla developers with a bash shell for many development tasks, including, but not limited to: build, version control (hg/git), editing (vim/emacs/etc), running scripts, etc.
I've been running latest (over stable) for some time with no problems. Given this will be a non-default (at least initially) mozilla-build console emulator, perhaps we don't need to be as cautious? :-)
ANSI sequences... I mean Esc seq., like \[[0m. Mostly for colors, but not limited. Why I asking. ConEmu has internal processor of ANSI, and it is required injecting of ConEmuHk.dll in each process running in console. It works pretty fast in latest, but may cause lags (sometimes large) in stable. On the other hand, bash does not send Esc sequences to console, when it see Windows and process them internally. As for latest version... Usually, I recommend last alpha or preview, But there are some problems, which I'm going to fix soon. For example, buffer (scrolling) not returns after Vim exiting.
(In reply to Avi Halachmi (:avih) from comment #35) > ... with ConEmu I was to get more colors (... after some configuration). Vlad asked me to post my color configuration, but apparently with latest ConEmu and latest mozilla-build, my specific config is no longer required, as it appears to work cleanly out of the box*. * I have ConEmu > config > In-console options > InjectConEmuHk and ANSI X3.64... checked, but not sure if that's the ConEmu default. I tried a fresh copy of ConEmu and they're still checked, but ConEmu could have taken the config from my other instance.
It is easy to check ConEmu defaults Just run: ConEmu.exe /reset And it will starts like new fresh configuration It will not save default settings until you press "Save settings" in the settings dialog, with only exception of options from "Fast configuration" dialog.
BTW, may be interesting. If you put near to ConEmu.exe any ConEmu.ico file - it will be used as default icon.
(In reply to Maximus from comment #39) > It is easy to check ConEmu defaults > Just run: ConEmu.exe /reset With clean ConEmu config, (with or without InjectConEmuHk), and without any custom .hgrc file, colors work out of the box for hg (tried log, diff, qdiff, qser), for ls (after aliasing to |ls --color=auto|) and for vim. Colors don't work with mach, however, and IIRC they never did on windows with or without ConEmu.
See Also: → 767434
(In reply to Ted Mielczarek [:ted.mielczarek] (post-vacation backlog) from comment #26) > I'm fine with shipping whatever in the package. Making something the default > is a higher bar, in that it can't break the interactive modes of frequently > used programs. If something has solved the latter then let's go for it. winpty solves this problem for mintty, in my testing.
fwiw, I've switched entirely to ConEmu, with no problems and only nice delightful surprises, such as the automatic Progress tracking (http://code.google.com/p/conemu-maximus5/wiki/Progress).
I'm not actively working on this. But someone else should. Perhaps RyanVM - our new MozillaBuild overlord - can be talked into being every Windows developer's hero.
Assignee: gps → nobody
Status: ASSIGNED → NEW
So Vlad has been using ConEmu with some success for a while now. Can we switch to it please? So that I don't feel like stabbing myself in the face every time I have to use Windows? ;-)
FWIW, I'm still using ConEmu happily. One thing I noticed, however, is that the inject dll hack can sometimes have a negative effect. The author mentions that it's possibly somewhat slower (haven't noticed this one myself), but I've experienced build errors while building mpv (the video player) with cygwin and a python waf build script - consistently and only when the dll hack injection was enabled. In Mozilla tasks I don't recall noticing any negative side effect of this dll hack injection, but to stay on the safe side, I suggest that if we add this, we disable it by default. So far I haven't seen noticed any deficiencies when it's disabled.
I've started to use ConEmu as well and am quite happy with it so far. Well, to be more precise, I cannot imagine ever going back to cmd.exe after this!
1. I can't remember any reported issues regarded to build problems. Have you reported them? 2. With injects disabled you will get certain MS bugs and crashes: http://code.google.com/p/conemu-maximus5/wiki/MicrosoftBugs
(regarded -> related)
(In reply to Maximus from comment #48) > 1. I can't remember any reported issues regarded to build problems. Have you > reported them? No. Took me way more than I needed to find out where do the build errors come from (only after I accidentally compiled outside of ConEmu and it suddenly completed), and I just wanted to get going. To reproduce it might be hard now, and even back then, the build environment and modifications would probably make it almost impossible for someone else to reproduce it. If I bump into it again, I will. > 2. With injects disabled you will get certain MS bugs and crashes: > http://code.google.com/p/conemu-maximus5/wiki/MicrosoftBugs Would this affect a bash environment with gnu bin utils and related tools? From my experience (win 7 and 8), MozillaBuild bash environment (and unrelated portable git bash environment), I don't think I noticed those bugs when the injection was disabled, and I also don't recall noticing any missing feature from ConEmu.
(In reply to Avi Halachmi (:avih) from comment #50) > ... only after I accidentally compiled outside of ConEmu and it > suddenly completed ... FWIW, I don't think it's necessarily ConEmu's fault. It might be some weird combination of cygwin and part of the modified toolchain or python which was buggy and susceptible to issues from the injection. Still, the less hacks, IMO the better. MS bugs are probably more widely known and easier to work around than bugs due the console's hacks to work around the MS bugs.
There's a mintty 1.0.3 package for MSYS 1.x. I intend to include it as part of the work in bug 791511.
Assignee: nobody → ryanvm
Blocks: MozillaBuild2.0
Depends on: 791511
minTTY + winpty works pretty well for the most part. I've run into a few issues that appear to be known issues with that setup (like pwd not being visible in the window title). Also, minTTY breaks the hg color extension (again). Manually setting the mode to win32 fixes it, but it'd probably be preferable to see if we can solve it upstream like we did previously in bug 545432. Right now, I think it's usable enough to ship by default, but I intend to make it easy to switch off if people aren't happy with the known issues it has. I'd really like to get hg fixed to Just Work, though.
Ryan and I talked on IRC. I'll look at fixing Mercurial upstream once I have my hands on a mintty-enabled MozillaBuild and have some time to look at it.
This is a checkpoint patch. Not ready for review yet. Test build: http://people.mozilla.org/~rvandermeulen/MozillaBuildSetup2.0.0pre-mintty.exe
Attachment #638007 - Attachment is obsolete: true
Updated patch, includes a minttyrc with some better defaults (which I unfortunately forgot to include in the recently-uploaded 2.0.0pre4 build).
Attachment #8566083 - Attachment is obsolete: true
Blocks: 1078116
I like the updated packages! :) I just tried http://people.mozilla.org/~rvandermeulen/MozillaBuildSetup2.0.0pre4.exe (with mintty as default), and I _think_ mintty is not very reliable when displaying colors (or possibly some ansi escape sequences?). I have my ~/.bash.rc file set LESSOPEN to use source-highlight to color source files, and with this 2.0pre4 package `less some/source.js` display it correctly when not using mintty (set USE_MINTTY=0) I.e. when bash runs within the native windows cmd.exe shell (or also within ConEmu), but with mintty it has weird formatting and navigation KB handling issues. If I disable the colors then less doesn't have neither of these issues in mintty. I didn't try it long enough to test if these issues also happen regardless of less. Also, I noticed that when I don't use mintty, and use start-shell-msvc13.bat to open directly within the current shell window, and then I `exit` or `logout` it closes the window - which is undesirable IMO. For mintty I guess it's OK to close the window on exit since a new window was opened for it and there's not really anything to return to after exit, but otherwise - exit should just exit the shell IMO and back to whatever shell it at was prior to starting it.
(In reply to Avi Halachmi (:avih) from comment #57) > Also, I noticed that when I don't use mintty, and use start-shell-msvc13.bat > to open directly within the current shell window, and then I `exit` or > `logout` it closes the window - which is undesirable IMO. ... I couldn't think of cases where the current code has advantages over the following change at `start-shell.bat` (which I tested in several cases and as far as I can tell works fine): Replace the first `EXIT` with `GOTO _DONE`, and replace the second `EXIT` with `:_DONE` This will let the batch end naturally, return to the caller batch `start-shell-MSVCXX.bat` which will also end naturally - and close the console window if executed from explorer or return to the previous shell otherwise.
(In reply to Avi Halachmi (:avih) from comment #57) > ... but with mintty it has weird formatting and navigation KB handling issues. Apparently it's not formatting. It's colors - it displays black instead of gray, so with parts of the code missing it looks like bad formatting on first glance. The arrow navigation still don't work on that case.
My brief experimentation with MinTTY revealed that there was no easy way to coerce Mercurial into doing proper color support. As mentioned elsewhere, MinTTY advertises a certain terminal (xterm IIRC) but doesn't support xterm's color sequences (or maybe I have this reversed). My understanding is this is a side-effect of MinTTY's architecture - it effectively operates as a magical pipe to a cmd.exe window or something like that. I suspect any tool doing color under MinTTY will need to be explicitly told which color mode to use, as autodetection will likely fail.
It seems any customization to mintty will be stored at /path/to/mozilla-build/msys/home/<username>/.minttyrc and will be lost if I removed the whole mozilla-build dir before installing a newer version.
Depends on: 1141841
Due to the various regressions noted in this bug and others, I've decided to not enable mintty by default in 2.0. I'll still include it for those who wish to use it and understand the pros and cons of doing so. I've also filed bug 1157744 to look into ConEmu as another possibility option for a default terminal. http://hg.mozilla.org/mozilla-build/rev/c447bc41cffe
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Depends on: 1182709
Product: mozilla.org → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: