1725895 - Switch mozilla-build to MSYS2, take 2

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, task)

Description

[Kagami Rosylight [:saschanaz] (they/them)]

It seems MSYS2 nearly just works, with the following steps:

1. Install MSYS2 by the official Windows x86-64 installer (msys2-x86_64-*) to C:/mozilla-build/msys2
2. Copy the files of msys/etc/profile.d to msys2/etc/profile.d
3. Replace $LOGNAME to $(logname) in profile-homedir.sh
4. Install some missing packages: pacman -S m4 diffutils
5. Add the following to start-shell.bat (or a separate start-shell-msys2.bat)

   SET MSYS=winsymlinks:nativestrict
   IF DEFINED MOZ_NO_RESET_PATH (
     SET MSYS2_PATH_TYPE=inherit
   )
   
   SET MOZILLABUILD_MSYS_BIN=%MOZILLABUILD%msys2\usr\bin
   IF DEFINED MOZILLABUILD_MSYS1 (
     SET MOZILLABUILD_MSYS_BIN=%MOZILLABUILD%msys\bin
   )
   

6. Replace all msys bin references to use MOZILLABUILD_MSYS_BIN if exists

Comment 1

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: WIP: Bug 1725895 - Use MOZILLABUILD_MSYS_BIN env variable if exists

Comment 2

[Kagami Rosylight [:saschanaz] (they/them)]

As above, I'd like to see MozillaBuild finally migrate to MSYS2, what do you think? (The attached patch only modifies m-c)

Comment 3

[Kagami Rosylight [:saschanaz] (they/them)]

This will finally allow us to update ssh binary.

Edit: I wonder we can default to MOZ_NO_RESET_PATH so that the bundled Windows OpenSSH binary can be used by default?
Edit2: Or add the path to that binary by default.

Comment 4

[:glob ✱]

This isn't something that I'm active on anymore; redirecting to Mitch and Glandium.

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Heh, was digging into this a couple months ago and reached some similar conclusions. I'm currently leaning towards removing MozillaBuild and having Mach internally manage an msys2 environment, but I'd like to do some outreach with our Windows devs first to address some concerns Glandium had when we chatted about it (notes doc here).

A couple questions though:

• Why is MSYS=winsymlinks:nativestrict needed?
• Why is MSYS2_PATH_TYPE=inherit needed?
• Why do you set MOZILLABUILD_MSYS_BIN to the old MozillaBuild if MOZILLABUILD_MSYS1 is defined?

Comment 6

[Kagami Rosylight [:saschanaz] (they/them)]

I'm currently leaning towards removing MozillaBuild and having Mach internally manage an msys2 environment, but I'd like to do some outreach with our Windows devs first to address some concerns Glandium had when we chatted about it (notes doc here).

👍. A new MozillaBuild release sounds great so that smaller group of people can try it first.

A couple questions though:

• Why is MSYS=winsymlinks:nativestrict needed?
• Why is MSYS2_PATH_TYPE=inherit needed?

Those are from the MSYS2 launcher:

1. The first one should enable symlink support for bug 1643072 (which requires Python >=3.8 while the MozillaBuild bundled version is still 3.7)
2. The second is to keep MOZ_NO_RESET_PATH behavior since seemingly MSYS2 resets the path by default. It's useful as it allows calling VSCode and GitHub from msys shell.

• Why do you set MOZILLABUILD_MSYS_BIN to the old MozillaBuild if MOZILLABUILD_MSYS1 is defined?

I thought it could allow a smoother migration to have msys1 and msys2 together for at least one release, but probably not needed, yeah.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Bugzilla, why won't you let me cancel locally-set status changes >:(
I'm picking this up for the new MB release :)

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

I've been able to squash together some MSYS2 packages and making a functioning environment, but I've got some notes so far:

• On first-start, msys2 prints some information and some confusing non-errors about permissions:

mkdir: cannot change permissions of '/dev/shm': Permission denied
mkdir: cannot change permissions of '/dev/mqueue': Permission denied
'C:\Windows\system32\drivers\etc\hosts' -> '/etc/hosts'
'C:\Windows\system32\drivers\etc\protocol' -> '/etc/protocols'
'C:\Windows\system32\drivers\etc\services' -> '/etc/services'
'C:\Windows\system32\drivers\etc\networks' -> '/etc/networks'
MozillaBuild Install Directory: C:\mozillabuild-stage\mozilla-build\

Mitch@mitch-home-windows-desktop MSYS ~
$

I've provided an upstream patch for the confusing non-errors.

• There's some funk around the different possible terminals: the main two being the Windows Command Prompt and the MSYS2-recommended one being MinTTY. In general, mintty behaves better with msys2-native (linux-y) software, while the Command Prompt is better at Windows-native software.
  • If using mintty and Windows-native Python, we need to use winpty to manage signals (otherwise Python thinks it's being run in a non-interactive environment). Unfortunately, CTRL-C doesn't work in ./mach bootstrap - additional investigation into why is needed.
  • If using mintty and MSYS Python, our existing code will break because the platform is changing (e.g.: platform.uname(), all of our "is windows" checks, etc). I really don't want to have to (significantly) change the tree to port to msys2.
    • Related: I'm guessing that MSYS Python will be slower than "windows-native" Python, which will make Mach and potentially the build slower in a significant way. That's just a guess, though
  • We can go back to using MSYS with the Command Prompt, as we do with MSYS1, but:
    • There's one bug with hg where color codes are erroneously printed, but that's workaroundable by changing TERM from MSYS2's preferred value of "xterm-256color" to MSYS1's preferred value of "cygwin" (which could cause other side-effects)
    • hg show work still doesn't work in Python 3 + Command Prompt. Investigation is needed to determine why.
    • Out of the different combinations here, the msys software we use is much more likely to behave within mintty than within the Command Prompt, so I'd prefer not to fall back to Command Prompt if possible.
• Mercurial 6.0 is coming out "soon" with "Lots of Windows fixes" (!), but I didn't find that it affected any of the issues listed above

Comment 11

[Molly Howell (she/her) (mostly inactive)]

I'd be curious about how Windows Terminal behaves; I'm used it on MSYS2 before and I don't recall any problems.

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

I'd be curious about how Windows Terminal behaves; I'm used it on MSYS2 before and I don't recall any problems.

In my (minimal!) testing on Windows Terminal, I didn't immediately run into any hurdles, but I don't think I ran Mach or hg with it yet.
FWIW, MozillaBuild will need to function on some Windows machines that don't yet have Windows Terminal installed (EDIT: looks like it was added in 1903, so we should be in the clear), so we still need to find a "fallback" configuration that behaves properly.

Depending on how this work goes, I'm leaning towards adding official docs to the MozillaBuild wiki explaining how to set it up with Windows Terminal.

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

Did some digging tonight that went deeper than I was expecting, but good news popped out so I'm happy. Here's the story:

1. The current MozillaBuild release combines MSYS with the Windows "Command Prompt" terminal to make the console we know and love, but it runs into issues (e.g.: CTRL-C signal handling, hg show doesn't work with Python 3, etc) with software because the combination is weird: software running on Windows expects Command Prompt + win32, or (to a lesser extent) MSYS/Cygwin + mintty/conemu/etc. Our unholy combination manages to consistently trod down a path of infrequent testing for each app it tries to run.
2. We were pushing towards using mintty in this bug, but ended up not doing so by default because there were issues running win32 software from mintty.
3. Later, after that ticket stopped moving, winpty was slowly evolving and maturing - it allowed you to do winpty $win32_command, which would automatically run the $win32_command in a hidden, "real" Windows terminal and implicitly sync the contents to-and-from the mintty/conemu/etc terminal. The bummer is that you had to remember to prefix each nontrivial win32 command with winpty. Additionally, there was a performance cost.
4. As the years ticked by, there was some consensus that cygwin/msys should automatically detect win32 executables and run them through winpty, though this never happened.
5. Then, along came ConPTY, a Windows feature that entirely removes the need to sync with a hidden Windows terminal. The missing piece was that cygwin/msys needed to integrate with ConPTY to leverage it.
6. But, since we're living in the future (feels good, doesn't it?), cygwin has implemented support in this revision, released it in version 3.1, and MSYS2 has picked it up automatically in this commit.
7. However, there was a missing piece that I was tripping over: why, in my earlier testing, was Python choking unless winpty was used? Shouldn't MSYS2 automatically be using ConPTY? Well, though Cygwin is using ConPTY by default, MSYS2 is making it an opt-in flag due to instability in Cygwin 3.1.4.
8. However, now that we're based on Cywin 3.2.0, the situation has improved, and (though testing is needed) it might be stable enough to enable by default for MozillaBuild.
9. By setting MSYS=enable_pcon, running MSYS2 + mintty, then running ./mach bootstrap with win32 Python, it appears to be working great so far! So, mission accomplished, all because we live in the future? Hopefully 🤞

TL;DR:

• ConPTY is good.
• MSYS2 supports ConPTY, but requires MSYS=enable_pcon to opt-in
• Additional testing is needed to verify that MSYS2's ConPTY support is sufficiently stable
• If anything, the upcoming Cygwin 3.3.0 might improve things even further, if we can dare to dream.

Comment 15

[Emilio Cobos Álvarez (:emilio)]

Another thing we should consider doing while we're overhauling this is considering to default to HGENCODING=utf-8. Otherwise for example I can't use my full name as the mercurial username. Tkhis should also make it more consistent with macOS and Linux...

Comment 16

[Mitchell Hentges [:mhentges] 🦀]

Tkhis should also make it more consistent with macOS and Linux...

Hmm, good call, though I was curious why this was different compared with macOS and Linux.
It turns out because hg fetches the locale from locale.getpreferredencoding(). On Windows, this is the ANSI code page (cp1252).

Note that MSYS2's Python's default locale is UTF-8. However, since we're using Win32 Python, the default is back to cp1252.

Related: we could opt into Python's UTF-8 mode, but this has risk of breaking Windows build scripts and, besides, default Python behaviour is still to use cp1252 - I don't want to add "set PYTHONUTF8=1 in your environment variables" to the Firefox-dev bootstrapping guide.

TL;DR: Opting into HGENCODING=utf-8 makes the most sense. When we support working on Firefox out-of-MozillaBuild, we'll need to document the usage of HGENCODING=utf-8 in the docs accordingly.

Comment 17

[Kagami Rosylight [:saschanaz] (they/them)]

I don't want to add "set PYTHONUTF8=1 in your environment variables" to the Firefox-dev bootstrapping guide.

Maybe mach can set it automatically (for internal use), can't it?

Comment 18

[Mitchell Hentges [:mhentges] 🦀]

Maybe mach can set it automatically (for internal use), can't it?

Mach can set it automatically for internal use, but there's still a lot of hg interaction happening outside Mach (hg commit, etc).

Comment 19

[Mitchell Hentges [:mhentges] 🦀]

As part of this work, I'm planning on "inheriting" the PATH by default, as opposed to the default behaviour of restricting the PATH down when MSYS2 initializes.

Comment 22

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Add support for MSYS2 MozillaBuild

Comment 23

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Port `--profile-command` to pure-Python r?nalexander

As part of this, the shell-script part of ./amch can be removed,
making it pure Python.

There's a change in --profile-command behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because so much of Mach has already been run before
CLI arguments are parsed in the Python process.

If a developer wants to profile Mach itself, they can manually run
python3 -m cProfile -o <file> ./mach ...

Comment 24

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Allow invoking Mach outside of MozillaBuild

Assuming that the MOZILLABUILD environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.

Note that MozillaBuild still needs to be installed, and the
MOZILLABUILD environment variable will have to be set.

Depends on D133928

Comment 25

[Phabricator Automation]

Comment on attachment 9255570 [details]
Bug 1725895: Allow invoking Mach outside of MozillaBuild

Revision D133936 was moved to bug 1740123. Setting attachment 9255570 [details] to obsolete.

Comment 26

[Mitchell Hentges [:mhentges] 🦀]

The first one should enable symlink support for bug 1643072 (which requires Python >=3.8 while the MozillaBuild bundled version is still 3.7)

Kagami, do you know if MSYS=winsymlinks:nativestrict still needed if Python is updated to Python 3.9?

Comment 27

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Shorten length of editbin command

With the upcoming MSYS2 migration, there's too many DLLs and
we go over the "Windows command-line length limit".

By making each path relative, we shorten it.

Comment 28

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Migrate to MSYS2

• Require the use of a "reference" MSYS2 installation
• Automatically determine the dependencies of the provided list of
  top-level package requirements.
• Use the "zstandard" Python package to extract ".zst" files because
  MozillaBuild 3.4 doesn't ship with "zstd"

The MSYS2 package population works as follows:

1. Communicate with MSYS2's package manager (pacman), have it update
   its cache with the package archives (that we'll use to populate our
   environment) and install the packages we need so we can query their
   dependencies.
2. Starting from the list of top-level packages, recursively discover
   the needed packages' names, versions, cache-archive path and
   source package url.
3. Once they're all discovered, extract them all to the "msys2" directory
   in MozillaBuild.

One breaking change handled here is the addition of the "/usr" directory.

Comment 29

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: [WILL-SQUASH] Remove MSYS packages

These are replaced by the MSYS2 package names defined
in packageit.py.

Comment 30

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Run MSYS2 package post-install scripts

This is needed for adopting the "ca-certificates" package,
whose post-install scripts actually populates ca-bundle.crt.

The post-install scripts are all run after all packages are
installed to avoid dependency issues. There's still risk
of dependency between post-install scripts, but this risk
is low and not worth carefully inverting the package dependency
tree.

Most of the other packages that have post-install scripts
just run "install-info".

Comment 31

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Use "ca-certificates" package

Replace our manually-updated "ca-bundle.crt" with the MSYS2
"ca-certificates" package.

Comment 32

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Use tar from external MSYS2 instead of old MozillaBuild

There's some MSYS-style tools that we need as part of packaging:
specifically, we need tar and xz for extracting MSYS2 packages.

Since we already depend on a valid external MSYS2 environment, let's
also lean on it for tar and xz.

This will bring us a step closer to being able to build MozillaBuild
without needing to install an older version first.

Comment 33

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Use MSYS2 distribution of wget

This allows us to leverage the MSYS2 package manager to
keep these tools up-to-date (and install dependencies, if
ever needed).

Version changes:

• [upx] 3.95 => 3.96: Bug fixes, no apparent breaking changes
• [wget] 1.20.3 => 1.21.2: Now requires gettext 0.19.3+,
  otherwise no significant changes that will affect us
• [yasm] Still 1.3.0.

Comment 34

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Define the "vi" shortcut inline

As a two-line file, it (subjectively) seems easier to manage
as an inline patch managed in packageit.py

Comment 35

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Remove obsolete inputrc

Most of the defaults were added back in 2006, and have been improved
within MSYS2 since. For example, completion-ignore-case works by
default, and the bell doesn't "ding" on-backspace or on-tab-completion.

Comment 36

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Remove custom minttyrc

MSYS2 ships some good mintty defaults these days, and if developers want
to do customization they can do it on their machine through mintty's
"Options" menu.

In other words: I think we should align ourselves as closely as possible
with "upstream" MSYS2.

Comment 37

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Remove `profile-homedir.sh`

The nsswitch.conf tweaks already change $HOME to point to
%USERPROFILE%.

Comment 38

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Simplify "profile" files

• Remove obsolete exports
• Don't print $VCDIR, $SDKDIR, $TOOLCHAIN because they aren't
  needed/used in regular MozillaBuild operation these days.
• Move EDITOR export to always occur, even if $MOZILLABUILD is not
  defined. This path should only really be triggerd if MozillaBuild's
  MSYS2 environment is activated outside of MozillaBuild - which is
  unlikely, but behaviour consistency is good.

Comment 39

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Clean up "msys\misc\" directory

The only remaining files in msys\misc\ were profile.d files, so rename
the directory to profile.d accordingly.

Change packageit.py to dynamically iterate through the directory.

Comment 40

[Kagami Rosylight [:saschanaz] (they/them)]

Kagami, do you know if MSYS=winsymlinks:nativestrict still needed if Python is updated to Python 3.9?

Retried on top of D133549 and it works without it 👍

Comment 41

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Remove texinfo, make, yasm

We're now bootstrapping mozmake, and texinfo doesn't appear to be
explicitly used in the Firefox build system anymore (there are .tex
files, but they appear to just be for 3rd-party code).

As of bug 1692940, we no longer need yasm.

Depends on D134466

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bfd5211060b4
Port `--profile-command` to pure-Python r=nalexander,glandium

Comment 44

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Add "--fetch-sources" support to packageit.py

If --fetch-sources is specified, then the associated source
archives for each installed package will be fetched and placed
into $stagedir\src.

Comment 45

[Marian-Vasile Laza]

Backed out 2 changesets (Bug 1725895, Bug 1740123) for causing build bustages.
Backout link
Push with failures
Failure Log

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/3be84d1c0723
Shorten length of editbin command r=glandium
https://hg.mozilla.org/mozilla-build/rev/d8773f6225dc
Remove texinfo, make, yasm r=glandium
https://hg.mozilla.org/mozilla-build/rev/e89b4156faf8
Migrate to MSYS2 r=glandium
https://hg.mozilla.org/mozilla-build/rev/339827415544
Use "ca-certificates" package r=glandium
https://hg.mozilla.org/mozilla-build/rev/59d5037d6c1a
Use tar from external MSYS2 instead of old MozillaBuild r=glandium
https://hg.mozilla.org/mozilla-build/rev/6b425005c543
Use MSYS2 distribution of wget r=glandium
https://hg.mozilla.org/mozilla-build/rev/cf4ec40277a6
Define the "vi" shortcut inline r=glandium
https://hg.mozilla.org/mozilla-build/rev/72ad018e3bc8
Remove obsolete inputrc r=glandium
https://hg.mozilla.org/mozilla-build/rev/e47c2011153d
Remove custom minttyrc r=glandium
https://hg.mozilla.org/mozilla-build/rev/8fa7defbb88a
Remove `profile-homedir.sh` r=glandium
https://hg.mozilla.org/mozilla-build/rev/55b3f0a6b12e
Simplify "profile" files r=glandium
https://hg.mozilla.org/mozilla-build/rev/8a3c7481e8c8
Clean up "msys\misc\" directory r=glandium

Comment 47

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd1960daa9e2
Port `--profile-command` to pure-Python r=nalexander,glandium

Comment 48

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/cd1960daa9e2

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/06d58374ab65
Add support for MSYS2 MozillaBuild r=glandium

Comment 50

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/06d58374ab65

Comment 51

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/b0de2a598e52
Add "--fetch-sources" support to packageit.py r=glandium

Comment 52

[Kagami Rosylight [:saschanaz] (they/them)]

Woohoo, great work Mitchell!!

Comment 53

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Fix inconsistent slashes in $PATH

Convert MOZILLABUILD path to unix-style before
adding it to the $PATH

Comment 54

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Shorten prompt

MSYS2 shows the $MSYSTEM ("MINGW64", "MSYS", "CLANG64", etc) in the
prompt. However, MozillaBuild will always use "MSYS" as the $MSYSTEM,
so showing it in the prompt is redundant and noisy.

This also makes the prompt equivalent to how it was with MozillaBuild
3.4.

Depends on D136674

Comment 55

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/7633f4aae8e7
Fix inconsistent slashes in $PATH r=glandium

Comment 56

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1725895: Add "wget" to MSYS2 package list

This was meant to be added as part of a previous patch (D134552),
but looking back it appears to be absent - perhaps a bad handling of a
merge conflict.

This re-adds wget to the environment, just as wget used to be
deployed with MozillaBuild in the last release.

Depends on D137579

Comment 57

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/10108a6f69dd
Shorten prompt r=glandium
https://hg.mozilla.org/mozilla-build/rev/fb5033bd6c6c
Add "wget" to MSYS2 package list r=glandium