1292715 - No double-click installer for new MSYS2-based Windows build environment

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Nathan Hakkakzadeh [:Nat]]

The current solution is rather complicated and we'd like it to be nearly seamless for prospective contributors to build mozilla on windows.

Comment 1

[Nathan Hakkakzadeh [:Nat]]

Attachment: Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Currently, it is written in [AutoIt](https://www.autoitscript.com/site/) which generates an exe.

Also, is a ConEmu preferences file which automatically points a newly installed ConEmu to the newly installed MSYS2.

Review commit: https://reviewboard.mozilla.org/r/69804/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69804/

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

::: python/mozboot/bin/ConEmu.xml:1
(Diff revision 1)
> +<?xml version="1.0" encoding="utf-8"?>

This is a gnarly file. Could you make the Mozilla-specific pieces more obvious?

Also, let's make a separate directory for this file. How about python/mozboot/support/.

::: python/mozboot/bin/bootstrap-msys2.au3:1
(Diff revision 1)
> +#include <InetConstants.au3>

We need a MPL 2.0 license header in this file.

::: python/mozboot/bin/bootstrap-msys2.au3:4
(Diff revision 1)
> +#include <InetConstants.au3>
> +#include <MsgBoxConstants.au3>
> +#include <WinAPIFiles.au3>
> +

Please add some documentation somewhere on how to turn this .au3 file into an exe.

::: python/mozboot/bin/bootstrap-msys2.au3:22
(Diff revision 1)
> +Func AutomateMSYS2Install($msysInstaller, $installLocation)
> +   Run($msysInstaller)
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("{ENTER}")
> +   Sleep(200)
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("^a")
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send($installLocation & "\msys64")
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("{ENTER}")
> +   Sleep(200)
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("^a")
> +   WinWaitActive("MSYS2 64bit Setup")
> +   Send("Moz MSYS2")
> +   Send("{ENTER}")
> +   WinWaitClose("MSYS2 64bit Setup")
> +   FileDelete($msysInstaller)
> +EndFunc

Ugh, this is a bit hacky :/

It's a shame msys2 doesn't provide a scriptable installer.

::: python/mozboot/bin/bootstrap-msys2.au3:47
(Diff revision 1)
> +
> +Local $installLocation = RequestInstallPath()
> +If $installLocation == "" Then
> +   Exit 1
> +EndIf
> +Local $msysInstaller = DownloadToTemp("http://repo.msys2.org/distrib/x86_64/msys2-x86_64-20160205.exe")

I'm a bit uneasy about downloading over plain text.

Did you look into verifying the SHA-256 of the downloaded file? If that's difficult, I can upload it to S3 or tooltool easily.

::: python/mozboot/bin/bootstrap-msys2.au3:52
(Diff revision 1)
> +Local $conEmuInstaller = DownloadToTemp("https://conemu.github.io/install2.ps1")
> +FileMove($conEmuInstaller, "install2.ps1")
> +; Install ConEmu
> +Local $psLaunch = "C:\Windows\System32\windowspowershell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Unrestricted "
> +RunWait($psLaunch & "set dst '" & $installLocation & "\ConEmu'; set ver 'stable'; set lnk 'Moz ConEmu'; set xml 'ConEmu.xml'; set run $TRUE; .\install2.ps1")
> +FileDelete("install2.ps1")

IIRC, running powershell.exe in unrestricted mode requires users to enable unrestricted mode first.

What happens when this runs on a fresh Windows install? Do users get the UAC prompt?

What happens if powershell is not installed? I /think/ it is installed on Windows 10 by default but not Windows 7 or 8. I could be wrong about that.

::: python/mozboot/bin/bootstrap-msys2.au3:56
(Diff revision 1)
> +; Download ConEmu
> +Local $conEmuInstaller = DownloadToTemp("https://conemu.github.io/install2.ps1")
> +FileMove($conEmuInstaller, "install2.ps1")
> +; Install ConEmu
> +Local $psLaunch = "C:\Windows\System32\windowspowershell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Unrestricted "
> +RunWait($psLaunch & "set dst '" & $installLocation & "\ConEmu'; set ver 'stable'; set lnk 'Moz ConEmu'; set xml 'ConEmu.xml'; set run $TRUE; .\install2.ps1")

This references ConEmu.xml but we never download that file.

Comment 3

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> Ugh, this is a bit hacky :/
> 
> It's a shame msys2 doesn't provide a scriptable installer.

If it were better, I wouldn't have to be using AutoIt. :(

> This references ConEmu.xml but we never download that file.

This is a bit of a chicken and the egg problem. The XML file needs to be download-able, so the XML file needs to be hosted somewhere.

Comment 4

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> If it were better, I wouldn't have to be using AutoIt. :(

There is another strategy. [Here](https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20160719.tar.xz/download) is a LZMA compressed directory that outputs what the MSYS2 installer does. The challenge though, is to find a stable direct link to the file (surprisingly difficult), download a small version of some console-based compressor that supports LZMA ([7zip?](http://7-zip.org/download.html)), and then decompress the MSYS2 to the destination, and then clean up all the temporary files.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> There is another strategy. [Here](https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20160719.tar.xz/download) is a LZMA compressed directory that outputs what the MSYS2 installer does. The challenge though, is to find a stable direct link to the file (surprisingly difficult), download a small version of some console-based compressor that supports LZMA ([7zip?](http://7-zip.org/download.html)), and then decompress the MSYS2 to the destination, and then clean up all the temporary files.

Since we've hard-coded the date of the release, we can always repackage the xz as a zip or something and host it on our own server. We may have to do this anyway for https.

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> Since we've hard-coded the date of the release, we can always repackage the xz as a zip or something and host it on our own server. We may have to do this anyway for https.

https://api.pub.build.mozilla.org/tooltool/sha512/f93a685c8a10abbd349cbef5306441ba235c4cbfba1cc000299e11b58f258e9953cbe23463515407925eeca94c3f5d8e5f637c95be387e620845efa43cdcb0c0 is a zip file containing the content of https://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20160719.tar.xz/download. Note: there is no `msys2` prefix in files in the archive. This should make filesystem layout a bit easier to manage.

The SHA-512 is in the URL.

Comment 7

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67218

> IIRC, running powershell.exe in unrestricted mode requires users to enable unrestricted mode first.
> 
> What happens when this runs on a fresh Windows install? Do users get the UAC prompt?
> 
> What happens if powershell is not installed? I /think/ it is installed on Windows 10 by default but not Windows 7 or 8. I could be wrong about that.

Users get a UAC prompt. [According to this Windows blog post](https://blogs.technet.microsoft.com/heyscriptingguy/2011/01/07/how-do-i-install-powershell-on-windows-7-and-other-questions/) Powershell is on Windows 7 by default, although it is somewhat hidden.

Comment 8

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/1-2/

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

Much better.

But I get fatal errors at the end of the install :/

::: python/mozboot/bin/bootstrap-msys2.vbs:51
(Diff revision 2)
> +Set FilesInZip = appShell.NameSpace(fso.GetAbsolutePathName("msys2.zip")).Items()
> +appShell.NameSpace(msysPath).CopyHere(FilesInZip)

This seems *really* slow when I run this locally. Is this the best way to uncompress a zip in powershell?

http://stackoverflow.com/questions/27768303/how-to-unzip-a-file-in-powershell suggests some alternatives.

::: python/mozboot/bin/bootstrap-msys2.vbs:79
(Diff revision 2)
> +shell.Run bashExecutable & " -l -c 'logout'", 1, true
> +shell.Run bashExecutable & " -l -c 'pacman -Syu --noconfirm wget mingw-w64-x86_64-python2-pip && logout'", 1, true
> +shell.Run conemuExecutable, 1, false
> +WScript.Sleep 1000
> +shell.SendKeys "export MOZ_WINDOWS_BOOTSTRAP=1 && cd "
> +shell.SendKeys installLocation
> +shell.SendKeys " && wget -q https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -O bootstrap.py && python bootstrap.py {ENTER}"

I get tons of "cannot open shared object file" errors at this stage.

I suspect PATH is wrong. This is possibly a result of me installing into a custom directory.

Comment 10

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

> This seems *really* slow when I run this locally. Is this the best way to uncompress a zip in powershell?
> 
> http://stackoverflow.com/questions/27768303/how-to-unzip-a-file-in-powershell suggests some alternatives.

Some of those solutions are specific to newer versions of Powershell. What's the oldest version of Windows we support?

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

> Some of those solutions are specific to newer versions of Powershell. What's the oldest version of Windows we support?

Let's say whatever is installable on Windows 7.

Comment 12

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67694

> I get tons of "cannot open shared object file" errors at this stage.
> 
> I suspect PATH is wrong. This is possibly a result of me installing into a custom directory.

I could not reproduce the exact error you mentioned, but I did find a related error which may fix your problem... will post my fix in the next review request.

Comment 13

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/2-3/

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review67828

Comment 15

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/3-4/

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68090

::: python/mozboot/bin/bootstrap-msys2.vbs:81
(Diff revision 4)
> +errorCode = shell.Run(bashExecutable & " -l -c 'logout'", 1, true)
> +If errorCode = 1 Then

Any non-0 exit code should be an error.

Comment 17

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/4-5/

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68114

This is very close.

::: python/mozboot/bin/bootstrap-msys2.vbs:55
(Diff revision 5)
> +Download "https://api.pub.build.mozilla.org/tooltool/sha512/f93a685c8a10abbd349cbef5306441ba235c4cbfba1cc000299e11b58f258e9953cbe23463515407925eeca94c3f5d8e5f637c95be387e620845efa43cdcb0c0", "msys2.zip"
> +Set FilesInZip = appShell.NameSpace(fso.GetAbsolutePathName("msys2.zip")).Items()
> +appShell.NameSpace(msysPath).CopyHere(FilesInZip)
> +' MSYS2 archive doesn't have tmp directory...
> +fso.CreateFolder(fso.BuildPath(msysPath, "tmp"))
> +fso.DeleteFile("msys2.zip")

We'll want to use a try..finally (or equivalent) block here to ensure msys2.zip is deleted from disk.

::: python/mozboot/bin/bootstrap-msys2.vbs:64
(Diff revision 5)
> +Download "https://conemu.github.io/install2.ps1", "install2.ps1"
> +conemuSettingsURI = "https://api.pub.build.mozilla.org/tooltool/sha512/9aa384ecc8025a974999e913c83064b3b797e05d19806e62ef558c8300e4c3f72967e9464ace59759f76216fc2fc66f338a1e5cdea3b9aa264529487f091d929"
> +' Run installer
> +shell.Run "powershell.exe -NoProfile -ExecutionPolicy Unrestricted set dst '" & conemuPath & "'; set ver 'stable'; set lnk 'Moz ConEmu'; set xml '" & conemuSettingsURI & "'; set run $FALSE; .\install2.ps1", 0, true
> +' Delete ConEmu installer
> +fso.DeleteFile("install2.ps1")

Ditto.

::: python/mozboot/bin/bootstrap-msys2.vbs:89
(Diff revision 5)
> +    WScript.Quit 1
> +End If
> +
> +errorCode = shell.Run(bashExecutable & " -l -c 'pacman -Syu --noconfirm wget mingw-w64-x86_64-python2-pip && logout'", 1, true)
> +If errorCode <> 0 Then
> +    MsgBox("Package update failed. Make sure you have internet access. If you think this is a bug, please file one in Bugzilla.")

Can we print a URL to file the bug at?

Comment 19

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/5-6/

Comment 20

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/6-7/

Comment 21

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68168

The error handling looks good!

One quick thing before this lands. Can we add some text to the path dialog saying something like: "When you click OK, we will download and extract a build environment to the directory specified. You should see various windows appear. Do NOT interact with them until one explicitly prompts you to continue."

Comment 22

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/7-8/

Comment 23

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68174

I want to r+ this. But it doesn't work on my machine :/

::: python/mozboot/bin/bootstrap-msys2.vbs:111
(Diff revision 8)
> +    WScript.Quit 1
> +End If
> +
> +errorCode = shell.Run(bashExecutable & " -l -c 'export MOZ_WINDOWS_BOOTSTRAP=1 && cd " & installLocation & " && wget -q https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py -O bootstrap.py && python bootstrap.py && logout {ENTER}'", 1, true)
> +If errorCode <> 0 Then
> +    MsgBox("Bootstrap failed. Make sure you have internet access. If you think this is a bug, please file one in Bugzilla https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Build%20Config")

I got this message when running this locally.

A window pops up real fast then I see the dialog box. I can try to track down the exact failure if you want...

Comment 24

[Nathan Hakkakzadeh [:Nat]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/69804/diff/8-9/

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8778514 [details]
Bug 1292715 - Added installer source code for MSYS2 and ConEmu.

https://reviewboard.mozilla.org/r/69804/#review68774

Looks good and works on my machine! Congratulations on figuring out the bootstrap invocation piece!

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0cd9b31c5729
Added installer source code for MSYS2 and ConEmu. r=gps

Comment 27

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0cd9b31c5729