1485228 - add wix binaries to existing fetch support for toolchains

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Kim Moir [:kmoir] ET]

- Get the tools for generating the MSI (which come from https://github.com/wixtoolset/wix3/releases/), hook that into the build system

i think the existing fetch task support should be good enough for those wix tools: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/fetch/toolchains.yml
2:22 PM they have release zips on their github releases page

Comment 1

[Kim Moir [:kmoir] ET]

So I assume the binary we want fetched from https://github.com/wixtoolset/wix3/releases/
is wix311-binaries.zip It contains the files that make up the WiX Toolset and is useful if you don't want to install the toolset -- for example, if you're just checking WiX into source control.

Looks like this is what we want from the example
https://www.codeproject.com/Tips/105638/A-quick-introduction-Create-an-MSI-installer-with

Comment 2

[Kim Moir [:kmoir] ET]

Attachment: wip patch

Adding binaries to the toolchains fetch.  The next step is to write a task that fetches and extracts the wix tools and runs the command on the installer.   Also, there is some work in the build system to add values for WIX variables.

Comment 3

[Kim Moir [:kmoir] ET]

So I have been doing a lot of reading about WiX, how to use it and looking at the taskgraph for how/when the artifacts are created to understand the scope of the work

In this comment the point is made that from a local dev environment it's not that important, the real issue is to get this working in CI so we can ship it
https://bugzilla.mozilla.org/show_bug.cgi?id=1475512#c6

Looking at the taskgraph, currently the 

repackage-win* task runs mach repackage installer to create target.installer.exe
repackage-signing-win* signs the target.installer.exe 

So the scope of the build team's work is to add something to mach repackage
like mach repackage msi which points to the installer and invokes a script to use the wix binaries to create the correct msi 

The releng team will take care of adding the tasks to the taskgraph to call this new mach repackage command so it is generated for every locale and for win32 and win64

Callek, does this accurately reflect the scope of work?

Comment 4

[Justin Wood (:Callek)]

(In reply to Kim Moir [:kmoir] ET Away until Sept 4 from comment #3)
> 
> Callek, does this accurately reflect the scope of work?

Yes, that aligns with my understanding

Comment 5

[Tom Prince [:tomprince]]

Looking to a future when we want cross-compiled windows builds, I was curious if there were cross-platform tools and discovered https://wiki.gnome.org/msitools which appears to support a subset of the wix language.

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: toolchains3.patch

updated patches which are not working.  I'm sure some of the syntax is wrong and I'm not sure if this is the right approach

So my requirements are to fetch the wix binaries, which will be installed in the build for the repackage task to create the msi.  I have other patches in progress to create the msi as part of ./mach repackage msi.

I talked to several folks in the build channel who suggested 
add a `fetch` stanza to the build you want, point at your new fetch.

add it to the list of things in toolchains in the build task definition: https://dxr.mozilla.org/mozilla-central/source/taskcluster/ci/build/windows.yml#65

Do I need it to to the transforms?  

when I run  ./mach taskgraph target -p ~/Downloads/parameters-m-i.yml 

I get 

File "/Users/kmoir/hg/mozilla-inbound/taskcluster/taskgraph/transforms/use_fetches.py", line 45, in use_fetches
    config.kind, job['name'], fetch))
Exception: Missing fetch job for toolchain-windows-wix: fetch

Comment 7

[Tom Prince [:tomprince]]

Comment on attachment 9008860 [details] [diff] [review]
toolchains3.patch

Review of attachment 9008860 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/fetch/toolchains.yml
@@ +197,5 @@
> +wix-3.1.1:
> +  description: wix 3.1.1 binary release
> +  treeherder:
> +    symbol: wix3.1.5
> +  run:

It looks like you based this on an old checkout of mozilla-central. Newer versions don't have a `treeherder` stanza, and have a `fetch` stanza instead of a `run` stanza.

::: taskcluster/ci/build/windows.yml
@@ +208,5 @@
>          - win64-clang-cl
>          - win64-rust
>          - win64-rust-size
>          - win64-sccache
> +        - windows-wix

This probably wants to be in the repackage tasks (https://searchfox.org/mozilla-central/rev/dd965445ec47fbf3cee566eff93b301666bda0e1/taskcluster/taskgraph/transforms/repackage.py#333) rather than the build.

(Used https://searchfox.org/mozilla-central/source/taskcluster/ci/repackage/kind.yml and https://searchfox.org/mozilla-central/source/taskcluster/ci/repackage-l10n/kind.yml for example)

::: taskcluster/ci/toolchain/windows.yml
@@ +266,5 @@
> +        symbol: TL(wix)
> +        tier: 1
> +    worker-type: aws-provisioner-v1/gecko-{level}-b-win2012
> +    worker:
> +        max-run-time: 3600

You'll need to have a run stanza here.

Alternatively, you could move the fetch stanza to the jobs that use wix, rather than republishing it via a toolchain task. Note that it puts the files in a slightly different location that toolchain deps would put them. It is unfortunate that we have (at least) two different ways to get artifacts from one task into another (i.e. toolchains and fechts).

Comment 8

[Kim Moir [:kmoir] ET]

Attachment: bug1485228-new.patch

:tomprince Thanks for the suggestions I was able to get it to download the file

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bde8f29dfc9065f2fe3c179ab4cff7ffb3844fac&selectedJob=199207784

But it fails extracting it
https://taskcluster-artifacts.net/PwgZzPiCRUyMWoLopAnWzA/0/public/logs/live_backing.log

with
https://taskcluster-artifacts.net/PwgZzPiCRUyMWoLopAnWzA/0/public/logs/live_backing.log

so I have to investigate there.

I realize that this will be used in a new repackage task not a build task but my understanding is that Callek is working on that and I just wanted to be able to test my other patches on try with the binaries available as an interim step.

Comment 9

[Kim Moir [:kmoir] ET]

I have patches that work create the MSI installer on a aws loaner.  They are unfortunately on my laptop that I accidentally left in the Toronto airport and the loaner I have here in MD I can't access viscosity to connect to the loaner.  In any case, my first priority when I get my laptop back in Toronto and return to Ottawa is to upload patches for Callek to look at.

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: bug1485228mon.patch

patches in progress.  The work remaining is to add a toolchain build to get the wix variables instead of having it locally installed. I'll attach a screenshot of the parameters I used to run it.

Comment 11

[Kim Moir [:kmoir] ET]

Attachment: Screen Shot 2018-10-01 at 10.17.50 AM.png

Comment 12

[Justin Wood (:Callek)]

Kim,

Is it possible to set the args: `locale`, `arch`, and `version` from build sys during the repackage call?

I suspect it would be simplest from my end that way, and also help reduce accidents in the process. (If not I can try to make it work, but useful to know if we can)

Comment 13

[Kim Moir [:kmoir] ET]

Certainly, I will fix this.

Comment 14

[Kim Moir [:kmoir] ET]

I have been looking at your request to use the values for locale, version and arch from the build system instead of explicitly specifying them via the command when invoking ./mach repackage msi. Looking at the other repackage scripts such as the mar one, it unzips the executable to get this information. All of this information is stored in buildhub.json for the build in ci, but not locally. Also, my understanding is that you didn't want to run configure as part of repackage so we wouldn't have config.status to parse. I'm going to ask in the #build channel for suggestions on how to proceed.

Comment 15

[Kim Moir [:kmoir] ET]

actually I talked to Callek and he is going to implement this in his taskgraph work which seems to be proceeding nicely.

Comment 16

[Kim Moir [:kmoir] ET]

remaining work to do is to fetch the wix binaries locally for running the ./mach repackage msi command locally similar to what is done for Node but for windows only.  To be honest very few developers will ever use this command so it's not the highest priority.

Comment 17

[Justin Wood (:Callek)]

Attachment: Bug 1485228 - Add Wix toolchain.

Comment 18

[Justin Wood (:Callek)]

Attachment: Bug 1485228 - Add support for repackage msi

Comment 19

[Justin Wood (:Callek)]

Attachment: Bug 1485228 - Callek's fixes/hacks to repackage-msi mach command

Comment 20

[Kim Moir [:kmoir] ET]

Attachment: bug1485228.patch

new patches with more error checking for Callek to integrate into his patches

Comment 21

[Kim Moir [:kmoir] ET]

Attachment: bug1485228-new.patch

updated to allow candle and light binaries be passed in

Comment 22

[Kim Moir [:kmoir] ET]

Callek do you have any feedback on these patches.

One question I had was whether 
FETCH_DIR = os.path.abspath('../../' + os.environ['MOZ_FETCHES_DIR']) 

should only be checked if the path candle and light are not passed or if this should be removed altogether?

Comment 23

[Kim Moir [:kmoir] ET]

Attachment: bug1485228-new2.patch

Updated it to reflect that  FETCH_DIR from being hardcoded and instead if not passed in just use `candle.exe` and `light.exe` (let PATH find them)

removed todos since they don't apply anymore

Comment 24

[Kim Moir [:kmoir] ET]

Attachment: bug1485228-new3.patch

fixed nits in phabricator review

Comment 25

[Justin Wood (:Callek)]

Kim, attachment 9022272 [details] [diff] [review] doesn't seem to have any of the fixes mentioned in phab (the return or the /Launch... stuff) did I miss something?

Comment 26

[Kim Moir [:kmoir] ET]

Attachment: bug1485228new4.patch

Sorry about that Callek, I added the additional content and tested it on my windows loaner.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/56387de74ad3
Add Wix toolchain. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/56165ffc77b2
Add support for repackage msi r=mhowell

Comment 28

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/56387de74ad3
https://hg.mozilla.org/mozilla-central/rev/56165ffc77b2