1289641 - Setup a taskcluster job to build the MSVC tooltool package

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Mike Hommey [:glandium]]

We should have a taskcluster job that automatically triggers when the build/windows_toolchain.py script changes (like we have for gcc and clang), and the script should be updated to create the package from scratch, instead of relying on a specific MSVC and SDK being already installed.

Comment 1

[Gregory Szorc [:gps]]

This should be doable now that Microsoft has made scriptable Visual Studio installs possible and TaskCluster Windows support has arrived. I'd feel much better about this being implemented in TaskCluster: currently I generate the official zip archives on a VM on one of my machines :/

Comment 2

[Rob Lemley [:rjl]]

Attachment: WIP: Bug 1289641 - MSVC & Win10 SDK toolchain build script.

Heavily modified from the initial Github source.

Downloads a bunch of components using the same manifest files as the
Visual Studio installer. The list needs some tweaking.

Unpacks the various payloads into a temporary directory and then packages
what is needed into a ZIP file. The ZIP creation uses the same code as
windows_toolchain.py. The only change is that as the ZIP is created, path
names are lower-cased. mozconfig.vs2017 will need some changes as a result.

This is not finished by far. Some of it may be over-engineered (do we need
multiprocess downloads?). Some other cleanup and some comments are needed
as well.

Comment 3

[Rob Lemley [:rjl]]

Attachment: Bug 1289641 - MSVC and Windows SDK private toolchain. r=#firefox-build-system-reviewers

Script to download and package the necessary components of MS Visual Studio 2017
and the Windows 10 SDK. The script uses the same JSON manifests as the Visual Studio
Installer.

After downloading, the files are repacked with the same layout as the current
tooltool files. All paths are converted to lower case as well.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1289641 - Don't pull Visual Studio in tasks that don't use it.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1289641 - Vendor the vsdownload script.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1289641 - Setup a taskcluster task to build the vs2017 toolchain.

Its layout differs from the one in the tooltool archive for forward
compatibility with the upcoming (clang 15) -winsysroot flag.

There is probably more filtering we can do to somehow match what
windows_toolchain.py used to do, but we'll keep that for later.

Comment 7

[Mike Hommey [:glandium]]

Attachment: Bug 1289641 - Replace uses of the vs2017-15.9 tooltool manifest.

Now that we have a toolchain artifact with the same function, we can use
that instead.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1289641 - Stop depending on the tooltool manifest to distinguish whether to build a windows winchecksec.

Comment 9

[Mike Hommey [:glandium]]

Attachment: Bug 1289641 - Allow spaces in MIDL and FXC paths.

On CI, 8.3 filenames are not generated, so the canonicalization of
program paths in windows kits doesn't work. But the build system can
actually deal with spaces in the specific case of MIDL and FXC.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8b22a8ec44fe
Don't pull Visual Studio in tasks that don't use it. r=firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/2645367da7cb
Vendor the vsdownload script. r=firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/acbff13b91b3
Setup a taskcluster task to build the vs2017 toolchain. r=firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/0b16a05aa405
Stop depending on the tooltool manifest to distinguish whether to build a windows winchecksec. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/55b68a71e03a
Allow spaces in MIDL and FXC paths. r=firefox-build-system-reviewers,nalexander
https://hg.mozilla.org/integration/autoland/rev/575732020bd2
Replace uses of the vs2017-15.9 tooltool manifest. r=firefox-build-system-reviewers,mhentges

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0c1cedebf915
Fix lint failure. a=fix CLOSED TREE

Comment 12

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/8b22a8ec44fe
https://hg.mozilla.org/mozilla-central/rev/2645367da7cb
https://hg.mozilla.org/mozilla-central/rev/acbff13b91b3
https://hg.mozilla.org/mozilla-central/rev/0b16a05aa405
https://hg.mozilla.org/mozilla-central/rev/55b68a71e03a
https://hg.mozilla.org/mozilla-central/rev/575732020bd2
https://hg.mozilla.org/mozilla-central/rev/0c1cedebf915

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/3d13f772cf70
Port Bug 1289641 - Setup a taskcluster job to build the MSVC tooltool package.rs=bustage-fix

Comment 14

[Atila Butkovits]

Backed out for causing Python failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/b655a765e94dd2e40d6516f2ff38779772617144

Push where failures started: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=KNL8YNUxSZGBHeWjK8E-Ww.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=76ca62e02f7328ca4b20af6d4046ab97937c8a3e

Failure log: https://treeherder.mozilla.org/logviewer?job_id=374250219&repo=autoland&lineNumber=362

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3245a51260b5
Don't pull Visual Studio in tasks that don't use it. r=firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/45fa68c69405
Vendor the vsdownload script. r=firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/bd9465c7f360
Setup a taskcluster task to build the vs2017 toolchain. r=firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/9f294248c8c2
Stop depending on the tooltool manifest to distinguish whether to build a windows winchecksec. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/c07590c5a6e9
Allow spaces in MIDL and FXC paths. r=firefox-build-system-reviewers,nalexander
https://hg.mozilla.org/integration/autoland/rev/bae6e398d032
Replace uses of the vs2017-15.9 tooltool manifest. r=firefox-build-system-reviewers,mhentges

Comment 16

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/3245a51260b5
https://hg.mozilla.org/mozilla-central/rev/45fa68c69405
https://hg.mozilla.org/mozilla-central/rev/bd9465c7f360
https://hg.mozilla.org/mozilla-central/rev/9f294248c8c2
https://hg.mozilla.org/mozilla-central/rev/c07590c5a6e9
https://hg.mozilla.org/mozilla-central/rev/bae6e398d032