Closed Bug 1725895 Opened 3 years ago Closed 2 years ago

Switch mozilla-build to MSYS2, take 2

Categories

(Firefox Build System :: MozillaBuild, task)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: saschanaz, Assigned: mhentges)

References

(Regressed 1 open bug)

Details

(Keywords: leave-open)

Attachments

(19 files, 3 obsolete files)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review

It seems MSYS2 nearly just works, with the following steps:

  1. Install MSYS2 by the official Windows x86-64 installer (msys2-x86_64-*) to C:/mozilla-build/msys2
  2. Copy the files of msys/etc/profile.d to msys2/etc/profile.d
  3. Replace $LOGNAME to $(logname) in profile-homedir.sh
  4. Install some missing packages: pacman -S m4 diffutils
  5. Add the following to start-shell.bat (or a separate start-shell-msys2.bat)
    SET MSYS=winsymlinks:nativestrict
    IF DEFINED MOZ_NO_RESET_PATH (
      SET MSYS2_PATH_TYPE=inherit
    )
    
    SET MOZILLABUILD_MSYS_BIN=%MOZILLABUILD%msys2\usr\bin
    IF DEFINED MOZILLABUILD_MSYS1 (
      SET MOZILLABUILD_MSYS_BIN=%MOZILLABUILD%msys\bin
    )
    
  6. Replace all msys bin references to use MOZILLABUILD_MSYS_BIN if exists

As above, I'd like to see MozillaBuild finally migrate to MSYS2, what do you think? (The attached patch only modifies m-c)

Flags: needinfo?(glob)

This will finally allow us to update ssh binary.

Edit: I wonder we can default to MOZ_NO_RESET_PATH so that the bundled Windows OpenSSH binary can be used by default?
Edit2: Or add the path to that binary by default.

See Also: → 1261899, 1681538
Attachment #9236371 - Attachment description: WIP: Bug 1725895 - Point to MSYS2 if MOZILLABUILD_MSYS2 exists → WIP: Bug 1725895 - Use MOZILLABUILD_MSYS_BIN env variable if exists

This isn't something that I'm active on anymore; redirecting to Mitch and Glandium.

Flags: needinfo?(mhentges)
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(glob)
See Also: → 1417003

Heh, was digging into this a couple months ago and reached some similar conclusions. I'm currently leaning towards removing MozillaBuild and having Mach internally manage an msys2 environment, but I'd like to do some outreach with our Windows devs first to address some concerns Glandium had when we chatted about it (notes doc here).

A couple questions though:

  • Why is MSYS=winsymlinks:nativestrict needed?
  • Why is MSYS2_PATH_TYPE=inherit needed?
  • Why do you set MOZILLABUILD_MSYS_BIN to the old MozillaBuild if MOZILLABUILD_MSYS1 is defined?
Flags: needinfo?(mhentges)

I'm currently leaning towards removing MozillaBuild and having Mach internally manage an msys2 environment, but I'd like to do some outreach with our Windows devs first to address some concerns Glandium had when we chatted about it (notes doc here).

👍. A new MozillaBuild release sounds great so that smaller group of people can try it first.

A couple questions though:

  • Why is MSYS=winsymlinks:nativestrict needed?
  • Why is MSYS2_PATH_TYPE=inherit needed?

Those are from the MSYS2 launcher:

  1. The first one should enable symlink support for bug 1643072 (which requires Python >=3.8 while the MozillaBuild bundled version is still 3.7)
  2. The second is to keep MOZ_NO_RESET_PATH behavior since seemingly MSYS2 resets the path by default. It's useful as it allows calling VSCode and GitHub from msys shell.
  • Why do you set MOZILLABUILD_MSYS_BIN to the old MozillaBuild if MOZILLABUILD_MSYS1 is defined?

I thought it could allow a smoother migration to have msys1 and msys2 together for at least one release, but probably not needed, yeah.

Flags: needinfo?(mh+mozilla)
Blocks: 1739443
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED

Bugzilla, why won't you let me cancel locally-set status changes >:(
I'm picking this up for the new MB release :)

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee: nobody → mhentges
Status: REOPENED → ASSIGNED

I've been able to squash together some MSYS2 packages and making a functioning environment, but I've got some notes so far:

  • On first-start, msys2 prints some information and some confusing non-errors about permissions:
mkdir: cannot change permissions of '/dev/shm': Permission denied
mkdir: cannot change permissions of '/dev/mqueue': Permission denied
'C:\Windows\system32\drivers\etc\hosts' -> '/etc/hosts'
'C:\Windows\system32\drivers\etc\protocol' -> '/etc/protocols'
'C:\Windows\system32\drivers\etc\services' -> '/etc/services'
'C:\Windows\system32\drivers\etc\networks' -> '/etc/networks'
MozillaBuild Install Directory: C:\mozillabuild-stage\mozilla-build\

Mitch@mitch-home-windows-desktop MSYS ~
$

I've provided an upstream patch for the confusing non-errors.

  • There's some funk around the different possible terminals: the main two being the Windows Command Prompt and the MSYS2-recommended one being MinTTY. In general, mintty behaves better with msys2-native (linux-y) software, while the Command Prompt is better at Windows-native software.
    • If using mintty and Windows-native Python, we need to use winpty to manage signals (otherwise Python thinks it's being run in a non-interactive environment). Unfortunately, CTRL-C doesn't work in ./mach bootstrap - additional investigation into why is needed.
    • If using mintty and MSYS Python, our existing code will break because the platform is changing (e.g.: platform.uname(), all of our "is windows" checks, etc). I really don't want to have to (significantly) change the tree to port to msys2.
      • Related: I'm guessing that MSYS Python will be slower than "windows-native" Python, which will make Mach and potentially the build slower in a significant way. That's just a guess, though
    • We can go back to using MSYS with the Command Prompt, as we do with MSYS1, but:
      • There's one bug with hg where color codes are erroneously printed, but that's workaroundable by changing TERM from MSYS2's preferred value of "xterm-256color" to MSYS1's preferred value of "cygwin" (which could cause other side-effects)
      • hg show work still doesn't work in Python 3 + Command Prompt. Investigation is needed to determine why.
      • Out of the different combinations here, the msys software we use is much more likely to behave within mintty than within the Command Prompt, so I'd prefer not to fall back to Command Prompt if possible.
  • Mercurial 6.0 is coming out "soon" with "Lots of Windows fixes" (!), but I didn't find that it affected any of the issues listed above

I'd be curious about how Windows Terminal behaves; I'm used it on MSYS2 before and I don't recall any problems.

I'd be curious about how Windows Terminal behaves; I'm used it on MSYS2 before and I don't recall any problems.

In my (minimal!) testing on Windows Terminal, I didn't immediately run into any hurdles, but I don't think I ran Mach or hg with it yet.
FWIW, MozillaBuild will need to function on some Windows machines that don't yet have Windows Terminal installed (EDIT: looks like it was added in 1903, so we should be in the clear), so we still need to find a "fallback" configuration that behaves properly.

Depending on how this work goes, I'm leaning towards adding official docs to the MozillaBuild wiki explaining how to set it up with Windows Terminal.

Did some digging tonight that went deeper than I was expecting, but good news popped out so I'm happy. Here's the story:

  1. The current MozillaBuild release combines MSYS with the Windows "Command Prompt" terminal to make the console we know and love, but it runs into issues (e.g.: CTRL-C signal handling, hg show doesn't work with Python 3, etc) with software because the combination is weird: software running on Windows expects Command Prompt + win32, or (to a lesser extent) MSYS/Cygwin + mintty/conemu/etc. Our unholy combination manages to consistently trod down a path of infrequent testing for each app it tries to run.
  2. We were pushing towards using mintty in this bug, but ended up not doing so by default because there were issues running win32 software from mintty.
  3. Later, after that ticket stopped moving, winpty was slowly evolving and maturing - it allowed you to do winpty $win32_command, which would automatically run the $win32_command in a hidden, "real" Windows terminal and implicitly sync the contents to-and-from the mintty/conemu/etc terminal. The bummer is that you had to remember to prefix each nontrivial win32 command with winpty. Additionally, there was a performance cost.
  4. As the years ticked by, there was some consensus that cygwin/msys should automatically detect win32 executables and run them through winpty, though this never happened.
  5. Then, along came ConPTY, a Windows feature that entirely removes the need to sync with a hidden Windows terminal. The missing piece was that cygwin/msys needed to integrate with ConPTY to leverage it.
  6. But, since we're living in the future (feels good, doesn't it?), cygwin has implemented support in this revision, released it in version 3.1, and MSYS2 has picked it up automatically in this commit.
  7. However, there was a missing piece that I was tripping over: why, in my earlier testing, was Python choking unless winpty was used? Shouldn't MSYS2 automatically be using ConPTY? Well, though Cygwin is using ConPTY by default, MSYS2 is making it an opt-in flag due to instability in Cygwin 3.1.4.
  8. However, now that we're based on Cywin 3.2.0, the situation has improved, and (though testing is needed) it might be stable enough to enable by default for MozillaBuild.
  9. By setting MSYS=enable_pcon, running MSYS2 + mintty, then running ./mach bootstrap with win32 Python, it appears to be working great so far! So, mission accomplished, all because we live in the future? Hopefully 🤞

TL;DR:

  • ConPTY is good.
  • MSYS2 supports ConPTY, but requires MSYS=enable_pcon to opt-in
  • Additional testing is needed to verify that MSYS2's ConPTY support is sufficiently stable
  • If anything, the upcoming Cygwin 3.3.0 might improve things even further, if we can dare to dream.

Another thing we should consider doing while we're overhauling this is considering to default to HGENCODING=utf-8. Otherwise for example I can't use my full name as the mercurial username. Tkhis should also make it more consistent with macOS and Linux...

Tkhis should also make it more consistent with macOS and Linux...

Hmm, good call, though I was curious why this was different compared with macOS and Linux.
It turns out because hg fetches the locale from locale.getpreferredencoding(). On Windows, this is the ANSI code page (cp1252).

Note that MSYS2's Python's default locale is UTF-8. However, since we're using Win32 Python, the default is back to cp1252.

Related: we could opt into Python's UTF-8 mode, but this has risk of breaking Windows build scripts and, besides, default Python behaviour is still to use cp1252 - I don't want to add "set PYTHONUTF8=1 in your environment variables" to the Firefox-dev bootstrapping guide.

TL;DR: Opting into HGENCODING=utf-8 makes the most sense. When we support working on Firefox out-of-MozillaBuild, we'll need to document the usage of HGENCODING=utf-8 in the docs accordingly.

I don't want to add "set PYTHONUTF8=1 in your environment variables" to the Firefox-dev bootstrapping guide.

Maybe mach can set it automatically (for internal use), can't it?

Maybe mach can set it automatically (for internal use), can't it?

Mach can set it automatically for internal use, but there's still a lot of hg interaction happening outside Mach (hg commit, etc).

As part of this work, I'm planning on "inheriting" the PATH by default, as opposed to the default behaviour of restricting the PATH down when MSYS2 initializes.

Depends on: 1520515
Depends on: 1419922
Depends on: 951291
Depends on: 803701
Keywords: leave-open
Attachment #9254843 - Attachment description: WIP: Bug 1725895: [DEV] Add support for MSYS2 MozillaBuild → Bug 1725895: Add support for MSYS2 MozillaBuild

As part of this, the shell-script part of ./amch can be removed,
making it pure Python.

There's a change in --profile-command behaviour, though: it now only
profiles the specific command, rather than all of Mach.
This is because so much of Mach has already been run before
CLI arguments are parsed in the Python process.

If a developer wants to profile Mach itself, they can manually run
python3 -m cProfile -o <file> ./mach ...

Assuming that the MOZILLABUILD environment variable is set, allow
invoking Mach from non-MozillaBuild terminals.

Note that MozillaBuild still needs to be installed, and the
MOZILLABUILD environment variable will have to be set.

Depends on D133928

Comment on attachment 9255570 [details]
Bug 1725895: Allow invoking Mach outside of MozillaBuild

Revision D133936 was moved to bug 1740123. Setting attachment 9255570 [details] to obsolete.

Attachment #9255570 - Attachment is obsolete: true

The first one should enable symlink support for bug 1643072 (which requires Python >=3.8 while the MozillaBuild bundled version is still 3.7)

Kagami, do you know if MSYS=winsymlinks:nativestrict still needed if Python is updated to Python 3.9?

Flags: needinfo?(krosylight)

With the upcoming MSYS2 migration, there's too many DLLs and
we go over the "Windows command-line length limit".

By making each path relative, we shorten it.

  • Require the use of a "reference" MSYS2 installation
  • Automatically determine the dependencies of the provided list of
    top-level package requirements.
  • Use the "zstandard" Python package to extract ".zst" files because
    MozillaBuild 3.4 doesn't ship with "zstd"

The MSYS2 package population works as follows:

  1. Communicate with MSYS2's package manager (pacman), have it update
    its cache with the package archives (that we'll use to populate our
    environment) and install the packages we need so we can query their
    dependencies.
  2. Starting from the list of top-level packages, recursively discover
    the needed packages' names, versions, cache-archive path and
    source package url.
  3. Once they're all discovered, extract them all to the "msys2" directory
    in MozillaBuild.

One breaking change handled here is the addition of the "/usr" directory.

These are replaced by the MSYS2 package names defined
in packageit.py.

This is needed for adopting the "ca-certificates" package,
whose post-install scripts actually populates ca-bundle.crt.

The post-install scripts are all run after all packages are
installed to avoid dependency issues. There's still risk
of dependency between post-install scripts, but this risk
is low and not worth carefully inverting the package dependency
tree.

Most of the other packages that have post-install scripts
just run "install-info".

Replace our manually-updated "ca-bundle.crt" with the MSYS2
"ca-certificates" package.

There's some MSYS-style tools that we need as part of packaging:
specifically, we need tar and xz for extracting MSYS2 packages.

Since we already depend on a valid external MSYS2 environment, let's
also lean on it for tar and xz.

This will bring us a step closer to being able to build MozillaBuild
without needing to install an older version first.

This allows us to leverage the MSYS2 package manager to
keep these tools up-to-date (and install dependencies, if
ever needed).

Version changes:

  • [upx] 3.95 => 3.96: Bug fixes, no apparent breaking changes
  • [wget] 1.20.3 => 1.21.2: Now requires gettext 0.19.3+,
    otherwise no significant changes that will affect us
  • [yasm] Still 1.3.0.

As a two-line file, it (subjectively) seems easier to manage
as an inline patch managed in packageit.py

Most of the defaults were added back in 2006, and have been improved
within MSYS2 since. For example, completion-ignore-case works by
default, and the bell doesn't "ding" on-backspace or on-tab-completion.

MSYS2 ships some good mintty defaults these days, and if developers want
to do customization they can do it on their machine through mintty's
"Options" menu.

In other words: I think we should align ourselves as closely as possible
with "upstream" MSYS2.

The nsswitch.conf tweaks already change $HOME to point to
%USERPROFILE%.

  • Remove obsolete exports
  • Don't print $VCDIR, $SDKDIR, $TOOLCHAIN because they aren't
    needed/used in regular MozillaBuild operation these days.
  • Move EDITOR export to always occur, even if $MOZILLABUILD is not
    defined. This path should only really be triggerd if MozillaBuild's
    MSYS2 environment is activated outside of MozillaBuild - which is
    unlikely, but behaviour consistency is good.

The only remaining files in msys\misc\ were profile.d files, so rename
the directory to profile.d accordingly.

Change packageit.py to dynamically iterate through the directory.

Kagami, do you know if MSYS=winsymlinks:nativestrict still needed if Python is updated to Python 3.9?

Retried on top of D133549 and it works without it 👍

Flags: needinfo?(krosylight)
Attachment #9256637 - Attachment is obsolete: true

We're now bootstrapping mozmake, and texinfo doesn't appear to be
explicitly used in the Firefox build system anymore (there are .tex
files, but they appear to just be for 3rd-party code).

As of bug 1692940, we no longer need yasm.

Depends on D134466

Attachment #9256640 - Attachment description: Bug 1725895: Use MSYS2 distributions of upx, wget and yasm → Bug 1725895: Use MSYS2 distribution of wget
Attachment #9256639 - Attachment description: Bug 1725895: Use tools from external MSYS2 instead of old MozillaBuild → Bug 1725895: Use tar from external MSYS2 instead of old MozillaBuild
Attachment #9256647 - Attachment description: Bug 1725895: Rename "msys\misc\" directory → Bug 1725895: Clean up "msys\misc\" directory
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bfd5211060b4
Port `--profile-command` to pure-Python r=nalexander,glandium

If --fetch-sources is specified, then the associated source
archives for each installed package will be fetched and placed
into $stagedir\src.

Backed out 2 changesets (Bug 1725895, Bug 1740123) for causing build bustages.
Backout link
Push with failures
Failure Log

Flags: needinfo?(mhentges)
Flags: needinfo?(mhentges)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/3be84d1c0723
Shorten length of editbin command r=glandium
https://hg.mozilla.org/mozilla-build/rev/d8773f6225dc
Remove texinfo, make, yasm r=glandium
https://hg.mozilla.org/mozilla-build/rev/e89b4156faf8
Migrate to MSYS2 r=glandium
https://hg.mozilla.org/mozilla-build/rev/339827415544
Use "ca-certificates" package r=glandium
https://hg.mozilla.org/mozilla-build/rev/59d5037d6c1a
Use tar from external MSYS2 instead of old MozillaBuild r=glandium
https://hg.mozilla.org/mozilla-build/rev/6b425005c543
Use MSYS2 distribution of wget r=glandium
https://hg.mozilla.org/mozilla-build/rev/cf4ec40277a6
Define the "vi" shortcut inline r=glandium
https://hg.mozilla.org/mozilla-build/rev/72ad018e3bc8
Remove obsolete inputrc r=glandium
https://hg.mozilla.org/mozilla-build/rev/e47c2011153d
Remove custom minttyrc r=glandium
https://hg.mozilla.org/mozilla-build/rev/8fa7defbb88a
Remove `profile-homedir.sh` r=glandium
https://hg.mozilla.org/mozilla-build/rev/55b3f0a6b12e
Simplify "profile" files r=glandium
https://hg.mozilla.org/mozilla-build/rev/8a3c7481e8c8
Clean up "msys\misc\" directory r=glandium
Regressions: 1748761
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cd1960daa9e2
Port `--profile-command` to pure-Python r=nalexander,glandium
Attachment #9254843 - Attachment description: Bug 1725895: Add support for MSYS2 MozillaBuild → WIP: Bug 1725895: Add support for MSYS2 MozillaBuild
Attachment #9254843 - Attachment description: WIP: Bug 1725895: Add support for MSYS2 MozillaBuild → Bug 1725895: Add support for MSYS2 MozillaBuild
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/06d58374ab65
Add support for MSYS2 MozillaBuild r=glandium
Attachment #9236371 - Attachment is obsolete: true
Regressions: 1749702
Regressions: 1749681
No longer regressions: 1749702
Attachment #9257575 - Attachment description: Bug 1725895: Add "--fetch-sources" support to packageit.py → WIP: Bug 1725895: Add "--fetch-sources" support to packageit.py
Attachment #9257575 - Attachment description: WIP: Bug 1725895: Add "--fetch-sources" support to packageit.py → Bug 1725895: Add "--fetch-sources" support to packageit.py
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/b0de2a598e52
Add "--fetch-sources" support to packageit.py r=glandium
Status: ASSIGNED → RESOLVED
Closed: 3 years ago2 years ago
Resolution: --- → FIXED

Woohoo, great work Mitchell!!

Convert MOZILLABUILD path to unix-style before
adding it to the $PATH

Attachment #9259799 - Attachment description: WIP: Bug 1725895: Fix inconsistent slashes in $PATH → Bug 1725895: Fix inconsistent slashes in $PATH
Regressions: 1751500

MSYS2 shows the $MSYSTEM ("MINGW64", "MSYS", "CLANG64", etc) in the
prompt. However, MozillaBuild will always use "MSYS" as the $MSYSTEM,
so showing it in the prompt is redundant and noisy.

This also makes the prompt equivalent to how it was with MozillaBuild
3.4.

Depends on D136674

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/7633f4aae8e7
Fix inconsistent slashes in $PATH r=glandium

This was meant to be added as part of a previous patch (D134552),
but looking back it appears to be absent - perhaps a bad handling of a
merge conflict.

This re-adds wget to the environment, just as wget used to be
deployed with MozillaBuild in the last release.

Depends on D137579

Regressions: 1753749
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/10108a6f69dd
Shorten prompt r=glandium
https://hg.mozilla.org/mozilla-build/rev/fb5033bd6c6c
Add "wget" to MSYS2 package list r=glandium
Product: mozilla.org → Firefox Build System
See Also: → 1864183
See Also: 1864183
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: