1739443 - Release MozillaBuild 4.0

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, task, P2)

Description

[Mitchell Hentges [:mhentges] 🦀]

• Migrate from MSYS to MSYS2
  • Remove MSYS "path resetting"
    • Ensure that cinnabar doesn't need to be manually added to the MozillaBuild path
  • Update terminal, bash, etc (hopefully resolving signal mismanagement)
  • This might resolving coloring issues
• Update Python 3
  • Work around WinError when doing python3 -m venv
• Update hg to 6.0
• Update nsis
• Make nano the default instead of emacs
• Bump ssh (or, IMHO, use the Windows built-in ssh instead)
• Fix fzf breaking copy/paste
  • Shipping "MinTTY" will resolve this.

Comment 1

[Mike Hommey [:glandium]]

Time to remove the things we now bootstrap? (nsis, mozmake)

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Time to remove the things we now bootstrap? (nsis, mozmake)

Sounds great to me 👍

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Format `packageit.py` with black==21.10b0

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Support "Visual Studio Build Tools"

By default, when identifying Visual Studio, vswhere only looks for
a full installation (Enterprise/Professional/Community).
However, the simpler standalone "build tools" installations provide
the needed tools (editbin.exe), so include it in the vswhere search
filter.

Depends on D130801

Comment 5

[Phabricator Automation]

Comment on attachment 9250081 [details]
Bug 1739443: Format packageit.py with black==21.10b0

Revision D130801 was moved to bug 1740357. Setting attachment 9250081 [details] to obsolete.

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Don't include nsis or mozmake in package

mozmake and nsis are now fetching during dev environment
bootstrapping, and are no longer needed in MozillaBuild itself.

Note that NSIS can't be removed from the repo because it's needed
to package MozillaBuild itself.

Depends on D130807

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Ensure hg works well with Python 3 (hg show ... doesn't work with py3 with current MozillaBuild)

This issue was because we customize hg's behaviour with a mercurial.ini in MozillaBuild, but we were only putting it in the Python2 installation, but not Python3.
So, hg config pager.pager returns less <with options> with Python 2, but nothing with Python 3.
So, when running hg show work with Python 3, it:

• looks for the pager config, but finds nothing
• falls back to the default (more)
• looks for more in the PATH and finds C:\Windows\System32\more.com (!)
• Stops on this line during a os.dup2(...) operation after trying to run more.com

Bug created upstream, but of course we can happily work around it.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Hmm, the pager issue is worse than I thought, and blocks our usage of hg with Python 3. I've deferred that work to this ticket.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/6519a7bdb760
Don't include nsis or mozmake in package r=glandium

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Gets me every time.

Comment 11

[Mitchell Hentges [:mhentges] 🦀]

This might resolving coloring issues

Hmm, the root cause of the lack of colouring in, say, ./mach build output is because we use blessings, which doesn't work even if we install windows-curses.

However, there's a Windows-compatible fork called blessed which should be relatively easy to port to.

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Remove old nsis/mozmake files from existing installations

In pave-over installs (installations where old files were not removed)
the old nsis and mozmake files will remain unless manually removed.
Add manual deletion to the installer script.

Also, remove obsolete NSIS entry from the PATH

Depends on D132517

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Update Python to 3.9.9

macOS and some mainline Linux distros have been using Python 3.9+ for
months, so we should be safe bumping it in MozillaBuild.

Depends on D133542

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/64e03545e006
Remove old nsis/mozmake files from existing installations r=glandium
https://hg.mozilla.org/mozilla-build/rev/3c3fdca8aa06
Update Python to 3.9.9 r=glandium

Comment 15

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Shorten length of editbin command

With the upcoming MSYS2 migration, there's too many DLLs and
we go over the "Windows command-line length limit".

By making each path relative, we shorten it.

Comment 16

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Migrate to MSYS2

• Require the use of a "reference" MSYS2 installation
• Automatically determine the dependencies of the provided list of
  top-level package requirements.
• Add requirements.[in|text] files to track the zstandard dependency

Comment 17

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: [WILL-SQUASH] Remove MSYS packages

These are replaced by the MSYS2 package names defined
in packageit.py.

Depends on D134467

Comment 18

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Run MSYS2 package post-install scripts

This is needed for adopting the "ca-certificates" package,
whose post-install scripts actually populates ca-bundle.crt.

Most of the other packages that have post-install scripts
just run "install-info".

Comment 19

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Use "ca-certificates" package

Replace our manually-updated "ca-bundle.crt" with the MSYS2
"ca-certificates" package.

Comment 20

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Use tools from external MSYS2 instead of old MozillaBuild

There's some MSYS-style tools that we need as part of packaging:
specifically, we need tar and xz for extracting MSYS2 packages.

Since we already depend on a valid external MSYS2 environment, let's
also lean on it for tar and xz.

This will bring us a step closer to being able to build MozillaBuild
without needing to install an older version first.

Comment 21

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Install windows-curses so `hg` can be used interactively

We can remove the interface = text constraint once we provide
the curses implementation for Windows.

Comment 22

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Use MSYS2 distributions of upx, wget and yasm

Comment 23

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Define the "vi" shortcut inline

As a two-line file, it (subjectively) seems easier to manage
as an inline patch managed in packageit.py

Comment 24

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Remove obsolete inputrc

Most of the defaults were added back in 2006, and have been improved
within MSYS2 since. For example, completion-ignore-case works by
default, and the bell doesn't "ding" on-backspace or on-tab-completion.

Depends on D134559

Comment 25

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Remove custom minttyrc

MSYS2 ships some good mintty defaults these days, and if developers want
to do customization they can do it on their machine through mintty's
"Options" menu.

In other words: I think we should align ourselves as closely as possible
with "upstream" MSYS2.

Depends on D134567

Comment 26

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Remove `profile-homedir.sh`

The nsswitch.conf tweaks already change $HOME to point to
%USERPROFILE%.

Depends on D134568

Comment 27

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Simplify "profile" files

• Remove obsolete exports
• Don't print $VCDIR, $SDKDIR, $TOOLCHAIN because they aren't
  needed/used in regular MozillaBuild operation these days.
• Move EDITOR export to always occur, even if $MOZILLABUILD is not
  defined. This path should only really be triggerd if MozillaBuild's
  MSYS2 environment is activated outside of MozillaBuild - which is
  unlikely, but behaviour consistency is good.

Depends on D134569

Comment 28

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Embrace native Windows ssh

Rather than shipping OpenSSH and manually managing its agent, lean on
Windows' built-in SSH tooling which was released as part of Windows 10
1809.

Accordingly, since Windows' OpenSSH doesn't read MSYS2's
/etc/ssh_config, remove it. This should be ok for two reasons:

• Now, we're going to always be using an up-to-date SSH version
• Modern SSH versions have sufficiently-good defaults.

Note that this means that, for older Windows Server instances, they'll
need to manually install OpenSSH if they need ssh.

Depends on D134570

Comment 29

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Rename "msys\misc\" directory

The only remaining files in msys\misc\ were profile.d files, so rename
the directory to profile.d accordingly.

Change packageit.py to dynamically iterate through the directory.

Depends on D134571

Comment 30

[Phabricator Automation]

Comment on attachment 9256476 [details]
Bug 1739443: Shorten length of editbin command

Revision D134466 was moved to bug 1725895. Setting attachment 9256476 [details] to obsolete.

Comment 31

[Phabricator Automation]

Comment on attachment 9256478 [details]
Bug 1739443: Migrate to MSYS2

Revision D134467 was moved to bug 1725895. Setting attachment 9256478 [details] to obsolete.

Comment 32

[Phabricator Automation]

Comment on attachment 9256479 [details]
Bug 1739443: [WILL-SQUASH] Remove MSYS packages

Revision D134468 was moved to bug 1725895. Setting attachment 9256479 [details] to obsolete.

Comment 33

[Phabricator Automation]

Comment on attachment 9256602 [details]
Bug 1739443: Run MSYS2 package post-install scripts

Revision D134547 was moved to bug 1725895. Setting attachment 9256602 [details] to obsolete.

Comment 34

[Phabricator Automation]

Comment on attachment 9256604 [details]
Bug 1739443: Use "ca-certificates" package

Revision D134549 was moved to bug 1725895. Setting attachment 9256604 [details] to obsolete.

Comment 35

[Phabricator Automation]

Comment on attachment 9256607 [details]
Bug 1739443: Use tools from external MSYS2 instead of old MozillaBuild

Revision D134550 was moved to bug 1725895. Setting attachment 9256607 [details] to obsolete.

Comment 36

[Phabricator Automation]

Comment on attachment 9256610 [details]
Bug 1739443: Use MSYS2 distributions of upx, wget and yasm

Revision D134552 was moved to bug 1725895. Setting attachment 9256610 [details] to obsolete.

Comment 37

[Phabricator Automation]

Comment on attachment 9256618 [details]
Bug 1739443: Define the "vi" shortcut inline

Revision D134559 was moved to bug 1725895. Setting attachment 9256618 [details] to obsolete.

Comment 38

[Phabricator Automation]

Comment on attachment 9256626 [details]
Bug 1739443: Remove obsolete inputrc

Revision D134567 was moved to bug 1725895. Setting attachment 9256626 [details] to obsolete.

Comment 39

[Phabricator Automation]

Comment on attachment 9256627 [details]
Bug 1739443: Remove custom minttyrc

Revision D134568 was moved to bug 1725895. Setting attachment 9256627 [details] to obsolete.

Comment 40

[Phabricator Automation]

Comment on attachment 9256628 [details]
Bug 1739443: Remove profile-homedir.sh

Revision D134569 was moved to bug 1725895. Setting attachment 9256628 [details] to obsolete.

Comment 41

[Phabricator Automation]

Comment on attachment 9256629 [details]
Bug 1739443: Simplify "profile" files

Revision D134570 was moved to bug 1725895. Setting attachment 9256629 [details] to obsolete.

Comment 42

[Phabricator Automation]

Comment on attachment 9256630 [details]
Bug 1739443: Embrace native Windows ssh

Revision D134571 was moved to bug 1261899. Setting attachment 9256630 [details] to obsolete.

Comment 43

[Phabricator Automation]

Comment on attachment 9256631 [details]
Bug 1739443: Rename "msys\misc" directory

Revision D134572 was moved to bug 1725895. Setting attachment 9256631 [details] to obsolete.

Comment 44

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Use "nano" as the default editor

Most power-users will likely want to use another editor, but now it will
be easier for new developers to get started.

Depends on D134572

Comment 45

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Configure mercurial to use UTF-8

Rather than falling back to the Windows system encoding, use UTF-8.
This provides benefits such as allowing accents in author names.

Depends on D134575

Comment 46

[Phabricator Automation]

Comment on attachment 9256648 [details]
Bug 1739443: Use "nano" as the default editor

Revision D134575 was moved to bug 1419922. Setting attachment 9256648 [details] to obsolete.

Comment 47

[Phabricator Automation]

Comment on attachment 9256649 [details]
Bug 1739443: Configure mercurial to use UTF-8

Revision D134576 was moved to bug 1419922. Setting attachment 9256649 [details] to obsolete.

Comment 48

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Prune mercurial.ini

Most of our "opinionated setup" is now happening via
hg configwizard in version-control-tools. This affects
extensions, some diff config, and other pieces.

The only remaining config that should still be shipped
with MozillaBuild's mercurial.ini should be
MozillaBuild-specific.

Comment 49

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Add "--fetch-sources" support to packageit.py

If --fetch-sources is specified, then the associated source
archives for each installed package will be fetched and placed
into $stagedir\src.

Comment 50

[Phabricator Automation]

Comment on attachment 9257574 [details]
Bug 1739443: Add "--fetch-sources" support to packageit.py

Revision D135091 was moved to bug 1725895. Setting attachment 9257574 [details] to obsolete.

Comment 51

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/077af10e2aa5
Install windows-curses so `hg` can be used interactively r=glandium
https://hg.mozilla.org/mozilla-build/rev/b983f777e887
Use "nano" as the default editor r=glandium
https://hg.mozilla.org/mozilla-build/rev/63a48b9fd120
Prune mercurial.ini r=glandium,sheehan

Comment 52

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Have nano automatically wrap at 72 chars for VCS

When writing commit messages for Firefox, it's recommended to manually
wrap lines that are more than roughly 72 characters long.

Use the nano "guidestrip" feature to make this easier for developers to
track.

Depends on D135091

Comment 53

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Bump version to 4.0pre0

🎉

Depends on D135187

Comment 54

[Mitchell Hentges [:mhentges] 🦀]

Status update:

• I'm encountering a bug right now where Python output that wraps over multiple lines is awful to copy-paste, because each line-wrap is copied as an explicit newline character that has to be massaged out. Gross >:(
• I might be able to move hg to Python 3 after all, because there's a potential workaround to its blocking issue.
  • It depends whether I can scope the workaround (set the PYTHONLEGACYWINDOWSSTDIO environment variable) to only apply to hg, as I don't want other Python code to be impacted.
• Preparing the launch documentation (migrations, workarounds, confetti, etc)

Comment 55

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/5734401c77de
Have nano automatically wrap at 72 chars for VCS r=glandium

Comment 56

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1739443: Populate pip packages, even if installed on host

Without this, mercurial isn't set up in MozillaBuild if the host
has it pip-installed elsewhere.

Depends on D136675

Comment 57

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/f614d70eb851
Bump version to 4.0pre0 r=glandium

Comment 58

[Mitchell Hentges [:mhentges] 🦀]

Status update:

• All issues slated to be resolved for 4.0 have patches on Phabricator. I've got roughly one more review to go before they can all be landed
• After being landed, I'm going to internally provide a "pre-release" of MozillaBuild 4.0. I'll let this simmer for roughly a week.
• Once any significant regressions detected in the pre-release (if any) have been resolved, MozillaBuild 4.0 itself will be publicly released.

Update: pre-release was published internally on March 4th, going to let it sit for a week or two before doing the full release.

Comment 59

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/60be1bb84ee7
Populate pip packages, even if installed on host r=glandium

Comment 60

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/bed13e443cc2
Bump version to 4.0pre1

Comment 61

[Mitchell Hentges [:mhentges] 🦀]

Status update:

• Two internal prereleases have been internally shipped - the first found some issues with our VCS integration
• Assuming no serious issues crop up, I'm planning on releasing MozillaBuild 4.0 early next week.

Comment 62

[Mitchell Hentges [:mhentges] 🦀]

MozillaBuild 4.0 has been released. 🎉