Closed Bug 1739443 Opened 2 years ago Closed 2 years ago

Release MozillaBuild 4.0

Categories

(Firefox Build System :: MozillaBuild, task, P2)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mhentges, Assigned: mhentges)

References

(Regressed 2 open bugs)

Details

(Keywords: leave-open)

Attachments

(9 files, 18 obsolete files)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
Priority: -- → P2

Time to remove the things we now bootstrap? (nsis, mozmake)

Time to remove the things we now bootstrap? (nsis, mozmake)

Sounds great to me 👍

Depends on: 1739717
Summary: Release new MozillaBuild version → Release MozillaBuild 4.0
Depends on: 1740357

By default, when identifying Visual Studio, vswhere only looks for
a full installation (Enterprise/Professional/Community).
However, the simpler standalone "build tools" installations provide
the needed tools (editbin.exe), so include it in the vswhere search
filter.

Depends on D130801

Comment on attachment 9250081 [details]
Bug 1739443: Format packageit.py with black==21.10b0

Revision D130801 was moved to bug 1740357. Setting attachment 9250081 [details] to obsolete.

Attachment #9250081 - Attachment is obsolete: true

mozmake and nsis are now fetching during dev environment
bootstrapping, and are no longer needed in MozillaBuild itself.

Note that NSIS can't be removed from the repo because it's needed
to package MozillaBuild itself.

Depends on D130807

Attachment #9250082 - Attachment is obsolete: true
Assignee: nobody → mhentges
Status: NEW → ASSIGNED
Depends on: 1725895

Ensure hg works well with Python 3 (hg show ... doesn't work with py3 with current MozillaBuild)

This issue was because we customize hg's behaviour with a mercurial.ini in MozillaBuild, but we were only putting it in the Python2 installation, but not Python3.
So, hg config pager.pager returns less <with options> with Python 2, but nothing with Python 3.
So, when running hg show work with Python 3, it:

  • looks for the pager config, but finds nothing
  • falls back to the default (more)
  • looks for more in the PATH and finds C:\Windows\System32\more.com (!)
  • Stops on this line during a os.dup2(...) operation after trying to run more.com

Bug created upstream, but of course we can happily work around it.

Hmm, the pager issue is worse than I thought, and blocks our usage of hg with Python 3. I've deferred that work to this ticket.

See Also: → 1743185
Component: General → MozillaBuild
Product: Firefox Build System → mozilla.org
Version: unspecified → other
Depends on: 1697833
Depends on: 1743192

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/6519a7bdb760
Don't include nsis or mozmake in package r=glandium

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED

Gets me every time.

Status: RESOLVED → REOPENED
Keywords: leave-open
Resolution: FIXED → ---

This might resolving coloring issues

Hmm, the root cause of the lack of colouring in, say, ./mach build output is because we use blessings, which doesn't work even if we install windows-curses.

However, there's a Windows-compatible fork called blessed which should be relatively easy to port to.

See Also: → 1744838
Depends on: 1261899
Depends on: 1744583

In pave-over installs (installations where old files were not removed)
the old nsis and mozmake files will remain unless manually removed.
Add manual deletion to the installer script.

Also, remove obsolete NSIS entry from the PATH

Depends on D132517

macOS and some mainline Linux distros have been using Python 3.9+ for
months, so we should be safe bumping it in MozillaBuild.

Depends on D133542

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/64e03545e006
Remove old nsis/mozmake files from existing installations r=glandium
https://hg.mozilla.org/mozilla-build/rev/3c3fdca8aa06
Update Python to 3.9.9 r=glandium

With the upcoming MSYS2 migration, there's too many DLLs and
we go over the "Windows command-line length limit".

By making each path relative, we shorten it.

Attached file Bug 1739443: Migrate to MSYS2 (obsolete) —
  • Require the use of a "reference" MSYS2 installation
  • Automatically determine the dependencies of the provided list of
    top-level package requirements.
  • Add requirements.[in|text] files to track the zstandard dependency

These are replaced by the MSYS2 package names defined
in packageit.py.

Depends on D134467

Attachment #9256476 - Attachment description: WIP: Bug 1739443: Shorten length of editbin command → Bug 1739443: Shorten length of editbin command
Attachment #9256478 - Attachment description: WIP: Bug 1739443: Migrate to MSYS2 → Bug 1739443: Migrate to MSYS2
Attachment #9256479 - Attachment description: WIP: Bug 1739443: [WILL-SQUASH] Remove MSYS packages → Bug 1739443: [WILL-SQUASH] Remove MSYS packages

This is needed for adopting the "ca-certificates" package,
whose post-install scripts actually populates ca-bundle.crt.

Most of the other packages that have post-install scripts
just run "install-info".

Attachment #9256602 - Attachment description: WIP: Bug 1739443: Run MSYS2 package post-install scripts → Bug 1739443: Run MSYS2 package post-install scripts

Replace our manually-updated "ca-bundle.crt" with the MSYS2
"ca-certificates" package.

Attachment #9256604 - Attachment description: WIP: Bug 1739443: Use "ca-certificates" package → Bug 1739443: Use "ca-certificates" package

There's some MSYS-style tools that we need as part of packaging:
specifically, we need tar and xz for extracting MSYS2 packages.

Since we already depend on a valid external MSYS2 environment, let's
also lean on it for tar and xz.

This will bring us a step closer to being able to build MozillaBuild
without needing to install an older version first.

Attachment #9256607 - Attachment description: WIP: Bug 1739443: Use tools from external MSYS2 instead of old MozillaBuild → Bug 1739443: Use tools from external MSYS2 instead of old MozillaBuild

We can remove the interface = text constraint once we provide
the curses implementation for Windows.

Attachment #9256608 - Attachment description: WIP: Bug 1739443: Install windows-curses so `hg` can be used interactively → Bug 1739443: Install windows-curses so `hg` can be used interactively
Attachment #9256610 - Attachment description: WIP: Bug 1739443: Use MSYS2 distributions of upx, wget and yasm TODO desc → WIP: Bug 1739443: Use MSYS2 distributions of upx, wget and yasm
Attachment #9256610 - Attachment description: WIP: Bug 1739443: Use MSYS2 distributions of upx, wget and yasm → Bug 1739443: Use MSYS2 distributions of upx, wget and yasm

As a two-line file, it (subjectively) seems easier to manage
as an inline patch managed in packageit.py

Attachment #9256618 - Attachment description: WIP: Bug 1739443: Define the "vi" shortcut inline → Bug 1739443: Define the "vi" shortcut inline
Attached file Bug 1739443: Remove obsolete inputrc (obsolete) —

Most of the defaults were added back in 2006, and have been improved
within MSYS2 since. For example, completion-ignore-case works by
default, and the bell doesn't "ding" on-backspace or on-tab-completion.

Depends on D134559

Attached file Bug 1739443: Remove custom minttyrc (obsolete) —

MSYS2 ships some good mintty defaults these days, and if developers want
to do customization they can do it on their machine through mintty's
"Options" menu.

In other words: I think we should align ourselves as closely as possible
with "upstream" MSYS2.

Depends on D134567

The nsswitch.conf tweaks already change $HOME to point to
%USERPROFILE%.

Depends on D134568

Attached file Bug 1739443: Simplify "profile" files (obsolete) —
  • Remove obsolete exports
  • Don't print $VCDIR, $SDKDIR, $TOOLCHAIN because they aren't
    needed/used in regular MozillaBuild operation these days.
  • Move EDITOR export to always occur, even if $MOZILLABUILD is not
    defined. This path should only really be triggerd if MozillaBuild's
    MSYS2 environment is activated outside of MozillaBuild - which is
    unlikely, but behaviour consistency is good.

Depends on D134569

Rather than shipping OpenSSH and manually managing its agent, lean on
Windows' built-in SSH tooling which was released as part of Windows 10
1809.

Accordingly, since Windows' OpenSSH doesn't read MSYS2's
/etc/ssh_config, remove it. This should be ok for two reasons:

  • Now, we're going to always be using an up-to-date SSH version
  • Modern SSH versions have sufficiently-good defaults.

Note that this means that, for older Windows Server instances, they'll
need to manually install OpenSSH if they need ssh.

Depends on D134570

The only remaining files in msys\misc\ were profile.d files, so rename
the directory to profile.d accordingly.

Change packageit.py to dynamically iterate through the directory.

Depends on D134571

Attachment #9256631 - Attachment description: Bug 1739443: Rename source "profile.d" file directory → Bug 1739443: Rename "msys\misc\" directory

Comment on attachment 9256476 [details]
Bug 1739443: Shorten length of editbin command

Revision D134466 was moved to bug 1725895. Setting attachment 9256476 [details] to obsolete.

Attachment #9256476 - Attachment is obsolete: true

Comment on attachment 9256478 [details]
Bug 1739443: Migrate to MSYS2

Revision D134467 was moved to bug 1725895. Setting attachment 9256478 [details] to obsolete.

Attachment #9256478 - Attachment is obsolete: true

Comment on attachment 9256479 [details]
Bug 1739443: [WILL-SQUASH] Remove MSYS packages

Revision D134468 was moved to bug 1725895. Setting attachment 9256479 [details] to obsolete.

Attachment #9256479 - Attachment is obsolete: true

Comment on attachment 9256602 [details]
Bug 1739443: Run MSYS2 package post-install scripts

Revision D134547 was moved to bug 1725895. Setting attachment 9256602 [details] to obsolete.

Attachment #9256602 - Attachment is obsolete: true

Comment on attachment 9256604 [details]
Bug 1739443: Use "ca-certificates" package

Revision D134549 was moved to bug 1725895. Setting attachment 9256604 [details] to obsolete.

Attachment #9256604 - Attachment is obsolete: true

Comment on attachment 9256607 [details]
Bug 1739443: Use tools from external MSYS2 instead of old MozillaBuild

Revision D134550 was moved to bug 1725895. Setting attachment 9256607 [details] to obsolete.

Attachment #9256607 - Attachment is obsolete: true

Comment on attachment 9256610 [details]
Bug 1739443: Use MSYS2 distributions of upx, wget and yasm

Revision D134552 was moved to bug 1725895. Setting attachment 9256610 [details] to obsolete.

Attachment #9256610 - Attachment is obsolete: true

Comment on attachment 9256618 [details]
Bug 1739443: Define the "vi" shortcut inline

Revision D134559 was moved to bug 1725895. Setting attachment 9256618 [details] to obsolete.

Attachment #9256618 - Attachment is obsolete: true

Comment on attachment 9256626 [details]
Bug 1739443: Remove obsolete inputrc

Revision D134567 was moved to bug 1725895. Setting attachment 9256626 [details] to obsolete.

Attachment #9256626 - Attachment is obsolete: true

Comment on attachment 9256627 [details]
Bug 1739443: Remove custom minttyrc

Revision D134568 was moved to bug 1725895. Setting attachment 9256627 [details] to obsolete.

Attachment #9256627 - Attachment is obsolete: true

Comment on attachment 9256628 [details]
Bug 1739443: Remove profile-homedir.sh

Revision D134569 was moved to bug 1725895. Setting attachment 9256628 [details] to obsolete.

Attachment #9256628 - Attachment is obsolete: true

Comment on attachment 9256629 [details]
Bug 1739443: Simplify "profile" files

Revision D134570 was moved to bug 1725895. Setting attachment 9256629 [details] to obsolete.

Attachment #9256629 - Attachment is obsolete: true

Comment on attachment 9256630 [details]
Bug 1739443: Embrace native Windows ssh

Revision D134571 was moved to bug 1261899. Setting attachment 9256630 [details] to obsolete.

Attachment #9256630 - Attachment is obsolete: true

Comment on attachment 9256631 [details]
Bug 1739443: Rename "msys\misc" directory

Revision D134572 was moved to bug 1725895. Setting attachment 9256631 [details] to obsolete.

Attachment #9256631 - Attachment is obsolete: true

Most power-users will likely want to use another editor, but now it will
be easier for new developers to get started.

Depends on D134572

Rather than falling back to the Windows system encoding, use UTF-8.
This provides benefits such as allowing accents in author names.

Depends on D134575

Comment on attachment 9256648 [details]
Bug 1739443: Use "nano" as the default editor

Revision D134575 was moved to bug 1419922. Setting attachment 9256648 [details] to obsolete.

Attachment #9256648 - Attachment is obsolete: true
Attachment #9256648 - Attachment is obsolete: false

Comment on attachment 9256649 [details]
Bug 1739443: Configure mercurial to use UTF-8

Revision D134576 was moved to bug 1419922. Setting attachment 9256649 [details] to obsolete.

Attachment #9256649 - Attachment is obsolete: true
No longer depends on: 1739717

Most of our "opinionated setup" is now happening via
hg configwizard in version-control-tools. This affects
extensions, some diff config, and other pieces.

The only remaining config that should still be shipped
with MozillaBuild's mercurial.ini should be
MozillaBuild-specific.

Depends on: 1744838
Depends on: 1748047
See Also: 1744838
Depends on: 1748531

If --fetch-sources is specified, then the associated source
archives for each installed package will be fetched and placed
into $stagedir\src.

Comment on attachment 9257574 [details]
Bug 1739443: Add "--fetch-sources" support to packageit.py

Revision D135091 was moved to bug 1725895. Setting attachment 9257574 [details] to obsolete.

Attachment #9257574 - Attachment is obsolete: true
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/077af10e2aa5
Install windows-curses so `hg` can be used interactively r=glandium
https://hg.mozilla.org/mozilla-build/rev/b983f777e887
Use "nano" as the default editor r=glandium
https://hg.mozilla.org/mozilla-build/rev/63a48b9fd120
Prune mercurial.ini r=glandium,sheehan

When writing commit messages for Firefox, it's recommended to manually
wrap lines that are more than roughly 72 characters long.

Use the nano "guidestrip" feature to make this easier for developers to
track.

Depends on D135091

Depends on: 1748762
Depends on: 1748753

🎉

Depends on D135187

Attachment #9257748 - Attachment description: Bug 1739443: Use nano with 80-char "guidestrip" with VCS → WIP: Bug 1739443: Have nano automatically wrap at 72 chars for VCS
Attachment #9257771 - Attachment description: Bug 1739443: Bump MozillaBuild version to 4.0 → WIP: Bug 1739443: Bump version to 4.0pre0
Attachment #9257748 - Attachment description: WIP: Bug 1739443: Have nano automatically wrap at 72 chars for VCS → Bug 1739443: Have nano automatically wrap at 72 chars for VCS
Attachment #9257771 - Attachment description: WIP: Bug 1739443: Bump version to 4.0pre0 → Bug 1739443: Bump version to 4.0pre0

Status update:

  • I'm encountering a bug right now where Python output that wraps over multiple lines is awful to copy-paste, because each line-wrap is copied as an explicit newline character that has to be massaged out. Gross >:(
  • I might be able to move hg to Python 3 after all, because there's a potential workaround to its blocking issue.
    • It depends whether I can scope the workaround (set the PYTHONLEGACYWINDOWSSTDIO environment variable) to only apply to hg, as I don't want other Python code to be impacted.
  • Preparing the launch documentation (migrations, workarounds, confetti, etc)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/5734401c77de
Have nano automatically wrap at 72 chars for VCS r=glandium
Depends on: 1743185
See Also: 1743185
Depends on: 1751500

Without this, mercurial isn't set up in MozillaBuild if the host
has it pip-installed elsewhere.

Depends on D136675

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/f614d70eb851
Bump version to 4.0pre0 r=glandium
Depends on: 1751526
Blocks: 1753063
Depends on: 1753259
Depends on: 1753749
Depends on: 1753751
Depends on: 1754290

Status update:

  • All issues slated to be resolved for 4.0 have patches on Phabricator. I've got roughly one more review to go before they can all be landed
  • After being landed, I'm going to internally provide a "pre-release" of MozillaBuild 4.0. I'll let this simmer for roughly a week.
  • Once any significant regressions detected in the pre-release (if any) have been resolved, MozillaBuild 4.0 itself will be publicly released.

Update: pre-release was published internally on March 4th, going to let it sit for a week or two before doing the full release.

Depends on: 1757813
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/mozilla-build/rev/60be1bb84ee7
Populate pip packages, even if installed on host r=glandium
Depends on: 1759047
Depends on: 1759547, 1758777
Depends on: 1759894

Status update:

  • Two internal prereleases have been internally shipped - the first found some issues with our VCS integration
  • Assuming no serious issues crop up, I'm planning on releasing MozillaBuild 4.0 early next week.
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → FIXED
Regressions: 1761092
Regressions: 1761444
Product: mozilla.org → Firefox Build System
You need to log in before you can comment on or make changes to this bug.