Generate visual studio tooltool archive from Visual Studio C++ build tools installer

NEW
Unassigned

Status

()

Core
Build Config
2 years ago
8 days ago

People

(Reporter: gps, Unassigned)

Tracking

(Depends on: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

https://blogs.msdn.microsoft.com/vcblog/2016/03/31/announcing-the-official-release-of-the-visual-c-build-tools-2015/

We currently recommend doing the full VS2015 installer. We should be able to use the installer announced at the above URL to derive the tooltool archive for the VS2015 toolchain.
For VS 2017:
https://blogs.msdn.microsoft.com/vcblog/2016/11/16/introducing-the-visual-studio-build-tools/

Also neat:
https://docs.microsoft.com/en-us/visualstudio/install/use-command-line-parameters-to-install-visual-studio
Also, we should code this up as a TaskCluster task, if possible.

And I don't know if we can get deterministic installation since the installer may download the latest version on every invocation. We may have to use that "offline installation" support so we have known versions.
You're probably right. That's documented here:
https://docs.microsoft.com/en-us/visualstudio/install/create-an-offline-installation-of-visual-studio

Something like:
1) Download vs_buildtools.exe installer: https://aka.ms/vs/15/release/vs_buildtools.exe
2) Create offline installer: `vs_buildtools.exe --layout \path\to\offline --lang en-US --add Microsoft.VisualStudio.Workload.VCTools` (still working out the smallest set of things we need here, the list is documented at https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools ), package up this directory as an artifact somewhere. (We could automate this, or just have a script to do it and put the result in tooltool.)
3) Have a toolchain task download the offline installer artifact, and then run `vs_buildtools.exe --installPath output_dir --lang en-US --add Microsoft.VisualStudio.Workload.VCTools --quiet --norestart --wait`, then package `output_dir` as an artifact, which should be a usable toolchain.

I will take a crack at this, I've been playing with the installers etc.
Oh, the offline installation page mentions "Install the certificates (They are in the Certificates folder, which is in your Layout folder. Simply right-click each one to install it.)". Presumably we will have to figure out an automated way to do this. Also, I'm not sure if this whole process requires administrative privileges, which might be tricky in Taskcluster.
From the comments on that page, a PowerShell command to install the certificates:
Import-PfxCertificate -FilePath .\certificates\manifestCounterSignCertificates.p12 -CertStoreLocation Cert:\CurrentUser\My
I made a task to do #1 and 2, let's see if it works:
https://tools.taskcluster.net/task-inspector/#ePfVoR6ZQ9uEWOjvllehpQ/
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> I made a task to do #1 and 2, let's see if it works:
> https://tools.taskcluster.net/task-inspector/#ePfVoR6ZQ9uEWOjvllehpQ/

This timed out. I ran the commands locally and running the `vs_buildtools.exe` command displays a UAC prompt, so I guess that's not surprising.
I'm still working on this, I think I've got it sorted now. I've been working in a clean Win10 VM, and so far this is what I've done:

I was able to generate an offline installer using the following Powershell script:
Invoke-RestMethod -Uri https://aka.ms/vs/15/release/vs_buildtools.exe -Method Get -OutFile vs_buildtools.exe
.\vs_buildtools.exe --layout vs_buildtools_offline --lang en-US --all --passive --norestart --wait

The resulting directory is 3.3GB. Presumably I could make that smaller by including just the components we need in --add, but I  hadn't quite figured out what those were yet.

Once that's done, I ran the following PowerShell command (in the vs_buildtools_offline directory) to install all the certificates:
Get-ChildItem certificates | ForEach-Object { Import-PfxCertificate -FilePath $_.FullName -CertStoreLocation Cert:\CurrentUser\My }

Then the following command in cmd.exe to install just the components we need:
.\vs_buildtools.exe --add Microsoft.VisualStudio.Component.VC.CoreBuildTools --installPath %cd%\..\vs_buildtools_2017 --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows10SDK.15063.Desktop --add Microsoft.VisualStudio.Component.VC.ATL --passive --norestart --wait

This seems to install OK except it didn't actually install the Windows 10 SDK.
OK, if I give up on installing the latest Win10 SDK I can get it to work. I was able to install  Microsoft.VisualStudio.Component.Windows10SDK.14393, which is the same version of the SDK we're currently using with VS2015 in automation, so it should be fine. I stuck the install certs command together with the installer commandline in a single powershell script, so I'm running:

Get-ChildItem certificates | ForEach-Object { Import-PfxCertificate -FilePath $_.FullName -CertStoreLocation Cert:\CurrentUser\My }
.\vs_buildtools.exe --installPath $PWD\..\vs_buildtools_2017 --add Microsoft.VisualStudio.Component.VC.CoreBuildTools --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.VC.ATL --add Microsoft.VisualStudio.Component.Windows10SDK.14393 --passive --norestart --wait

On the plus side, having figured all this out I should be able to take a crack at bug 1123821.
Yeah, let's stick with the 14393 SDK for now. It shouldn't cause any problems.

FWIW I saw a rumbling on Twitter from a Google engineer that the 15063 SDK may have an issue with a specific header file or something.
(In reply to Gregory Szorc [:gps] from comment #2)
> Also, we should code this up as a TaskCluster task, if possible.

Can taskcluster tasks produce private artifacts?
(In reply to Mike Hommey [:glandium] from comment #11)
> Can taskcluster tasks produce private artifacts?

I don't know! dustin?
Flags: needinfo?(dustin)
It can, although the logs are public.  Any artifact without a `public/` prefix needs a scope to create it and a scope to read it.

https://docs.taskcluster.net/manual/devel/namespaces#artifacts would suggest using a `project/gecko/` prefix for this artifact.  But we would need to think carefully about the name and about distribution of the relevant scope.
Flags: needinfo?(dustin)
We already have VC2015 archives in tooltool, guarded by the scope that grants access to download private tooltool artifacts.
The underlying concern is the files can't be "distributed" because that would be in violation of Microsoft's software use licenses. If we're OK with non-public tooltool archives, we should be OK with TC artifacts with similar access policies.
Depends on: 1405408
Depends on: 1408458
You need to log in before you can comment on or make changes to this bug.