1270667 - Generate visual studio tooltool archive from Visual Studio C++ build tools installer

Status: RESOLVED DUPLICATE of bug 1289641

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

https://blogs.msdn.microsoft.com/vcblog/2016/03/31/announcing-the-official-release-of-the-visual-c-build-tools-2015/

We currently recommend doing the full VS2015 installer. We should be able to use the installer announced at the above URL to derive the tooltool archive for the VS2015 toolchain.

Comment 1

[(not currently active) Ted Mielczarek]

For VS 2017:
https://blogs.msdn.microsoft.com/vcblog/2016/11/16/introducing-the-visual-studio-build-tools/

Also neat:
https://docs.microsoft.com/en-us/visualstudio/install/use-command-line-parameters-to-install-visual-studio

Comment 2

[Gregory Szorc [:gps]]

Also, we should code this up as a TaskCluster task, if possible.

And I don't know if we can get deterministic installation since the installer may download the latest version on every invocation. We may have to use that "offline installation" support so we have known versions.

Comment 3

[(not currently active) Ted Mielczarek]

You're probably right. That's documented here:
https://docs.microsoft.com/en-us/visualstudio/install/create-an-offline-installation-of-visual-studio

Something like:
1) Download vs_buildtools.exe installer: https://aka.ms/vs/15/release/vs_buildtools.exe
2) Create offline installer: `vs_buildtools.exe --layout \path\to\offline --lang en-US --add Microsoft.VisualStudio.Workload.VCTools` (still working out the smallest set of things we need here, the list is documented at https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools ), package up this directory as an artifact somewhere. (We could automate this, or just have a script to do it and put the result in tooltool.)
3) Have a toolchain task download the offline installer artifact, and then run `vs_buildtools.exe --installPath output_dir --lang en-US --add Microsoft.VisualStudio.Workload.VCTools --quiet --norestart --wait`, then package `output_dir` as an artifact, which should be a usable toolchain.

I will take a crack at this, I've been playing with the installers etc.

Comment 4

[(not currently active) Ted Mielczarek]

Oh, the offline installation page mentions "Install the certificates (They are in the Certificates folder, which is in your Layout folder. Simply right-click each one to install it.)". Presumably we will have to figure out an automated way to do this. Also, I'm not sure if this whole process requires administrative privileges, which might be tricky in Taskcluster.

Comment 5

[(not currently active) Ted Mielczarek]

From the comments on that page, a PowerShell command to install the certificates:
Import-PfxCertificate -FilePath .\certificates\manifestCounterSignCertificates.p12 -CertStoreLocation Cert:\CurrentUser\My

Comment 6

[(not currently active) Ted Mielczarek]

I made a task to do #1 and 2, let's see if it works:
https://tools.taskcluster.net/task-inspector/#ePfVoR6ZQ9uEWOjvllehpQ/

Comment 7

[(not currently active) Ted Mielczarek]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> I made a task to do #1 and 2, let's see if it works:
> https://tools.taskcluster.net/task-inspector/#ePfVoR6ZQ9uEWOjvllehpQ/

This timed out. I ran the commands locally and running the `vs_buildtools.exe` command displays a UAC prompt, so I guess that's not surprising.

Comment 8

[(not currently active) Ted Mielczarek]

I'm still working on this, I think I've got it sorted now. I've been working in a clean Win10 VM, and so far this is what I've done:

I was able to generate an offline installer using the following Powershell script:
Invoke-RestMethod -Uri https://aka.ms/vs/15/release/vs_buildtools.exe -Method Get -OutFile vs_buildtools.exe
.\vs_buildtools.exe --layout vs_buildtools_offline --lang en-US --all --passive --norestart --wait

The resulting directory is 3.3GB. Presumably I could make that smaller by including just the components we need in --add, but I  hadn't quite figured out what those were yet.

Once that's done, I ran the following PowerShell command (in the vs_buildtools_offline directory) to install all the certificates:
Get-ChildItem certificates | ForEach-Object { Import-PfxCertificate -FilePath $_.FullName -CertStoreLocation Cert:\CurrentUser\My }

Then the following command in cmd.exe to install just the components we need:
.\vs_buildtools.exe --add Microsoft.VisualStudio.Component.VC.CoreBuildTools --installPath %cd%\..\vs_buildtools_2017 --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows10SDK.15063.Desktop --add Microsoft.VisualStudio.Component.VC.ATL --passive --norestart --wait

This seems to install OK except it didn't actually install the Windows 10 SDK.

Comment 9

[(not currently active) Ted Mielczarek]

OK, if I give up on installing the latest Win10 SDK I can get it to work. I was able to install  Microsoft.VisualStudio.Component.Windows10SDK.14393, which is the same version of the SDK we're currently using with VS2015 in automation, so it should be fine. I stuck the install certs command together with the installer commandline in a single powershell script, so I'm running:

Get-ChildItem certificates | ForEach-Object { Import-PfxCertificate -FilePath $_.FullName -CertStoreLocation Cert:\CurrentUser\My }
.\vs_buildtools.exe --installPath $PWD\..\vs_buildtools_2017 --add Microsoft.VisualStudio.Component.VC.CoreBuildTools --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.VC.ATL --add Microsoft.VisualStudio.Component.Windows10SDK.14393 --passive --norestart --wait

On the plus side, having figured all this out I should be able to take a crack at bug 1123821.

Comment 10

[Gregory Szorc [:gps]]

Yeah, let's stick with the 14393 SDK for now. It shouldn't cause any problems.

FWIW I saw a rumbling on Twitter from a Google engineer that the 15063 SDK may have an issue with a specific header file or something.

Comment 11

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] from comment #2)
> Also, we should code this up as a TaskCluster task, if possible.

Can taskcluster tasks produce private artifacts?

Comment 12

[Gregory Szorc [:gps]]

(In reply to Mike Hommey [:glandium] from comment #11)
> Can taskcluster tasks produce private artifacts?

I don't know! dustin?

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

It can, although the logs are public.  Any artifact without a `public/` prefix needs a scope to create it and a scope to read it.

https://docs.taskcluster.net/manual/devel/namespaces#artifacts would suggest using a `project/gecko/` prefix for this artifact.  But we would need to think carefully about the name and about distribution of the relevant scope.

Comment 14

[(not currently active) Ted Mielczarek]

We already have VC2015 archives in tooltool, guarded by the scope that grants access to download private tooltool artifacts.

Comment 15

[Gregory Szorc [:gps]]

The underlying concern is the files can't be "distributed" because that would be in violation of Microsoft's software use licenses. If we're OK with non-public tooltool archives, we should be OK with TC artifacts with similar access policies.