Closed Bug 1292715 Opened 8 years ago Closed 8 years ago

No double-click installer for new MSYS2-based Windows build environment

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox51 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla51
Tracking Flags:
Tracking Status
firefox51 --- fixed

People

(Reporter: Nat, Assigned: Nat)

References

Details

Attachments

(1 file)

Bug 1292715 - Added installer source code for MSYS2 and ConEmu.
58 bytes, text/x-review-board-request
gps
: review+
Details 
The current solution is rather complicated and we'd like it to be nearly seamless for prospective contributors to build mozilla on windows.
Blocks: 1100925
Currently, it is written in [AutoIt](https://www.autoitscript.com/site/) which generates an exe.

Also, is a ConEmu preferences file which automatically points a newly installed ConEmu to the newly installed MSYS2.

Review commit: https://reviewboard.mozilla.org/r/69804/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69804/
Attachment #8778514 - Flags: review?(gps)
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

::: python/mozboot/bin/ConEmu.xml:1
(Diff revision 1)
> +<?xml version="1.0" encoding="utf-8"?>

This is a gnarly file. Could you make the Mozilla-specific pieces more obvious?

Also, let's make a separate directory for this file. How about python/mozboot/support/.

::: python/mozboot/bin/bootstrap-msys2.au3:1
(Diff revision 1)
> +#include <InetConstants.au3>

We need a MPL 2.0 license header in this file.

::: python/mozboot/bin/bootstrap-msys2.au3:4
(Diff revision 1)
> +#include <InetConstants.au3>
> +#include <MsgBoxConstants.au3>
> +#include <WinAPIFiles.au3>
> +

Please add some documentation somewhere on how to turn this .au3 file into an exe.

::: python/mozboot/bin/bootstrap-msys2.au3:22
(Diff revision 1)
> +Func AutomateMSYS2Install($msysInstaller, $installLocation)
> +   Run($msysInstaller)
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("{ENTER}")
> +   Sleep(200)
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("^a")
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send($installLocation & "\msys64")
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("{ENTER}")
> +   Sleep(200)
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("^a")
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("Moz MSYS2")
> +   Send("{ENTER}")
> +   WinWaitClose("MSYS2 64bit Setup")
> +   FileDelete($msysInstaller)
> +EndFunc

Ugh, this is a bit hacky :/

It's a shame msys2 doesn't provide a scriptable installer.

::: python/mozboot/bin/bootstrap-msys2.au3:47
(Diff revision 1)
> +
> +Local $installLocation = RequestInstallPath()
> +If $installLocation == "" Then
> +   Exit 1
> +EndIf
> +Local $msysInstaller = DownloadToTemp("http://repo.msys2.org/distrib/x86_64/msys2-x86_64-20160205.exe")

I'm a bit uneasy about downloading over plain text.

Did you look into verifying the SHA-256 of the downloaded file? If that's difficult, I can upload it to S3 or tooltool easily.

::: python/mozboot/bin/bootstrap-msys2.au3:52
(Diff revision 1)
> +Local $conEmuInstaller = DownloadToTemp("https://conemu.github.io/install2.ps1")
> +FileMove($conEmuInstaller, "install2.ps1")
> +; Install ConEmu
> +Local $psLaunch = "C:\Windows\System32\windowspowershell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Unrestricted "
> +RunWait($psLaunch & "set dst '" & $installLocation & "\ConEmu'; set ver 'stable'; set lnk 'Moz ConEmu'; set xml 'ConEmu.xml'; set run $TRUE; .\install2.ps1")
> +FileDelete("install2.ps1")

IIRC, running powershell.exe in unrestricted mode requires users to enable unrestricted mode first.

What happens when this runs on a fresh Windows install? Do users get the UAC prompt?

What happens if powershell is not installed? I /think/ it is installed on Windows 10 by default but not Windows 7 or 8. I could be wrong about that.

::: python/mozboot/bin/bootstrap-msys2.au3:56
(Diff revision 1)
> +; Download ConEmu
> +Local $conEmuInstaller = DownloadToTemp("https://conemu.github.io/install2.ps1")
> +FileMove($conEmuInstaller, "install2.ps1")
> +; Install ConEmu
> +Local $psLaunch = "C:\Windows\System32\windowspowershell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Unrestricted "
> +RunWait($psLaunch & "set dst '" & $installLocation & "\ConEmu'; set ver 'stable'; set lnk 'Moz ConEmu'; set xml 'ConEmu.xml'; set run $TRUE; .\install2.ps1")

This references ConEmu.xml but we never download that file.
Attachment #8778514 - Flags: review?(gps) → review-
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> Ugh, this is a bit hacky :/
> 
> It's a shame msys2 doesn't provide a scriptable installer.

If it were better, I wouldn't have to be using AutoIt. :(

> This references ConEmu.xml but we never download that file.

This is a bit of a chicken and the egg problem. The XML file needs to be download-able, so the XML file needs to be hosted somewhere.
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> If it were better, I wouldn't have to be using AutoIt. :(

There is another strategy. [Here](https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20160719.tar.xz/download) is a LZMA compressed directory that outputs what the MSYS2 installer does. The challenge though, is to find a stable direct link to the file (surprisingly difficult), download a small version of some console-based compressor that supports LZMA ([7zip?](http://7-zip.org/download.html)), and then decompress the MSYS2 to the destination, and then clean up all the temporary files.
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> There is another strategy. [Here](https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20160719.tar.xz/download) is a LZMA compressed directory that outputs what the MSYS2 installer does. The challenge though, is to find a stable direct link to the file (surprisingly difficult), download a small version of some console-based compressor that supports LZMA ([7zip?](http://7-zip.org/download.html)), and then decompress the MSYS2 to the destination, and then clean up all the temporary files.

Since we've hard-coded the date of the release, we can always repackage the xz as a zip or something and host it on our own server. We may have to do this anyway for https.
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> Since we've hard-coded the date of the release, we can always repackage the xz as a zip or something and host it on our own server. We may have to do this anyway for https.

https://api.pub.build.mozilla.org/tooltool/sha512/f93a685c8a10abbd349cbef5306441ba235c4cbfba1cc000299e11b58f258e9953cbe23463515407925eeca94c3f5d8e5f637c95be387e620845efa43cdcb0c0 is a zip file containing the content of https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20160719.tar.xz/download. Note: there is no `msys2` prefix in files in the archive. This should make filesystem layout a bit easier to manage.

The SHA-512 is in the URL.
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> IIRC, running powershell.exe in unrestricted mode requires users to enable unrestricted mode first.
> 
> What happens when this runs on a fresh Windows install? Do users get the UAC prompt?
> 
> What happens if powershell is not installed? I /think/ it is installed on Windows 10 by default but not Windows 7 or 8. I could be wrong about that.

Users get a UAC prompt. [According to this Windows blog post](https://blogs.technet.microsoft.com/heyscriptingguy/2011/01/07/how-do-i-install-powershell-on-windows-7-and-other-questions/) Powershell is on Windows 7 by default, although it is somewhat hidden.
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

Much better.

But I get fatal errors at the end of the install :/

::: python/mozboot/bin/bootstrap-msys2.vbs:51
(Diff revision 2)
> +Set FilesInZip = appShell.NameSpace(fso.GetAbsolutePathName("msys2.zip")).Items()
> +appShell.NameSpace(msysPath).CopyHere(FilesInZip)

This seems *really* slow when I run this locally. Is this the best way to uncompress a zip in powershell?

http://stackoverflow.com/questions/27768303/how-to-unzip-a-file-in-powershell suggests some alternatives.

::: python/mozboot/bin/bootstrap-msys2.vbs:79
(Diff revision 2)
> +shell.Run bashExecutable & " -l -c 'logout'", 1, true
> +shell.Run bashExecutable & " -l -c 'pacman -Syu --noconfirm wget mingw-w64-x86_64-python2-pip && logout'", 1, true
> +shell.Run conemuExecutable, 1, false
> +WScript.Sleep 1000
> +shell.SendKeys "export MOZ_WINDOWS_BOOTSTRAP=1 && cd "
> +shell.SendKeys installLocation
> +shell.SendKeys " && wget -q https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -O bootstrap.py && python bootstrap.py {ENTER}"

I get tons of "cannot open shared object file" errors at this stage.

I suspect PATH is wrong. This is possibly a result of me installing into a custom directory.
Attachment #8778514 - Flags: review?(gps) → review-
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

> This seems *really* slow when I run this locally. Is this the best way to uncompress a zip in powershell?
> 
> http://stackoverflow.com/questions/27768303/how-to-unzip-a-file-in-powershell suggests some alternatives.

Some of those solutions are specific to newer versions of Powershell. What's the oldest version of Windows we support?
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

> Some of those solutions are specific to newer versions of Powershell. What's the oldest version of Windows we support?

Let's say whatever is installable on Windows 7.
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

> I get tons of "cannot open shared object file" errors at this stage.
> 
> I suspect PATH is wrong. This is possibly a result of me installing into a custom directory.

I could not reproduce the exact error you mentioned, but I did find a related error which may fix your problem... will post my fix in the next review request.
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67828
Attachment #8778514 - Flags: review?(gps) → review-
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68090

::: python/mozboot/bin/bootstrap-msys2.vbs:81
(Diff revision 4)
> +errorCode = shell.Run(bashExecutable & " -l -c 'logout'", 1, true)
> +If errorCode = 1 Then

Any non-0 exit code should be an error.
Attachment #8778514 - Flags: review?(gps) → review-
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68114

This is very close.

::: python/mozboot/bin/bootstrap-msys2.vbs:55
(Diff revision 5)
> +Download "https://api.pub.build.mozilla.org/tooltool/sha512/f93a685c8a10abbd349cbef5306441ba235c4cbfba1cc000299e11b58f258e9953cbe23463515407925eeca94c3f5d8e5f637c95be387e620845efa43cdcb0c0", "msys2.zip"
> +Set FilesInZip = appShell.NameSpace(fso.GetAbsolutePathName("msys2.zip")).Items()
> +appShell.NameSpace(msysPath).CopyHere(FilesInZip)
> +' MSYS2 archive doesn't have tmp directory...
> +fso.CreateFolder(fso.BuildPath(msysPath, "tmp"))
> +fso.DeleteFile("msys2.zip")

We'll want to use a try..finally (or equivalent) block here to ensure msys2.zip is deleted from disk.

::: python/mozboot/bin/bootstrap-msys2.vbs:64
(Diff revision 5)
> +Download "https://conemu.github.io/install2.ps1", "install2.ps1"
> +conemuSettingsURI = "https://api.pub.build.mozilla.org/tooltool/sha512/9aa384ecc8025a974999e913c83064b3b797e05d19806e62ef558c8300e4c3f72967e9464ace59759f76216fc2fc66f338a1e5cdea3b9aa264529487f091d929"
> +' Run installer
> +shell.Run "powershell.exe -NoProfile -ExecutionPolicy Unrestricted set dst '" & conemuPath & "'; set ver 'stable'; set lnk 'Moz ConEmu'; set xml '" & conemuSettingsURI & "'; set run $FALSE; .\install2.ps1", 0, true
> +' Delete ConEmu installer
> +fso.DeleteFile("install2.ps1")

Ditto.

::: python/mozboot/bin/bootstrap-msys2.vbs:89
(Diff revision 5)
> +    WScript.Quit 1
> +End If
> +
> +errorCode = shell.Run(bashExecutable & " -l -c 'pacman -Syu --noconfirm wget mingw-w64-x86_64-python2-pip && logout'", 1, true)
> +If errorCode <> 0 Then
> +    MsgBox("Package update failed. Make sure you have internet access. If you think this is a bug, please file one in Bugzilla.")

Can we print a URL to file the bug at?
Attachment #8778514 - Flags: review?(gps) → review-
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68168

The error handling looks good!

One quick thing before this lands. Can we add some text to the path dialog saying something like: "When you click OK, we will download and extract a build environment to the directory specified. You should see various windows appear. Do NOT interact with them until one explicitly prompts you to continue."
Attachment #8778514 - Flags: review?(gps) → review-
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68174

I want to r+ this. But it doesn't work on my machine :/

::: python/mozboot/bin/bootstrap-msys2.vbs:111
(Diff revision 8)
> +    WScript.Quit 1
> +End If
> +
> +errorCode = shell.Run(bashExecutable & " -l -c 'export MOZ_WINDOWS_BOOTSTRAP=1 && cd " & installLocation & " && wget -q https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -O bootstrap.py && python bootstrap.py && logout {ENTER}'", 1, true)
> +If errorCode <> 0 Then
> +    MsgBox("Bootstrap failed. Make sure you have internet access. If you think this is a bug, please file one in Bugzilla https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Build%20Config")

I got this message when running this locally.

A window pops up real fast then I see the dialog box. I can try to track down the exact failure if you want...
Attachment #8778514 - Flags: review?(gps) → review-
Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68774

Looks good and works on my machine! Congratulations on figuring out the bootstrap invocation piece!
Attachment #8778514 - Flags: review?(gps) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/0cd9b31c5729
Added installer source code for MSYS2 and ConEmu. r=gps
https://hg.mozilla.org/mozilla-central/rev/0cd9b31c5729
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox51: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
Product: Core → Firefox Build System
Depends on: 1622405
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: