1199290 - Publish binaries to tooltool that are required for creating Windows™ worker types defined in taskcluster aws provisioner

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Pete Moore [:pmoore][:pete]]

Taskcluster tasks that run in Windows™ require various binary artifacts, such as Visual Studio, Windows SDK, MozillaBuild.

These should be published to tooltool for safekeeping.

This bug is to track the publication of all Windows binary artifacts that should be published to tooltool as required by Windows desktop builds of Firefox.

Comment 1

[Pete Moore [:pmoore][:pete]]

First one uploading now....

{
"size": 81352763, 
"visibility": "public", 
"digest": "03b4ca2bebede21a29f739165030bfc7058a461ffe38113452e976193e382d3ba6df8a48ac843b70429e23481e6327f43c86ffd88e4ce16263d072ef7e14e692", 
"algorithm": "sha512", 
"filename": "MozillaBuildSetup-2.0.0.exe"
}

Comment 2

[Pete Moore [:pmoore][:pete]]

Now installing Visual Studio 2013 Community Edition and Windows SDK 8.1 via chocolatey package manager.

E.g. see https://github.com/petemoore/generic-worker/blob/2a37d530e0ba8a3cb1870bf3a85ea0c58e8fb611/firefox.userdata#L33-L38

@Rob what would the steps be to get this internally hosted (or in s3)?

Comment 3

[Rob Thijssen [:grenade (EET/UTC+0300)]]

you can put nupkg files in:
- chocolatey packages: releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/chocolatey/nupkg/
- nuget packages: releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/nuget/nupkg/

these are served up from: http://releng-puppet2.srv.releng.scl3.mozilla.com/repos, but the repo behavior is still being developed and won't work as you'd expect from the choco/nuget clients.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

:+1: but a few concerns to add:
 - we shouldn't have automation hitting puppetmaster-hosted repos (and anyway the URL you gave isn't internet-accessible).
   instead, maybe it's time to set up choco.pub.build.mozilla.org or something like that?
 - there may be licensing issues hosting that particular choco package (since it's from MS)

Comment 5

[(not currently active) Ted Mielczarek]

catlee: This seems very related to things we were discussing last week re: AMI creation.

Comment 6

[Rob Thijssen [:grenade (EET/UTC+0300)]]

just want to link bug 1175133, where we're tracking choco/nuget repo stuff

Comment 7

[Pete Moore [:pmoore][:pete]]

Rob, is everything in tooltool now, or do we have stuff in the OCC manifests that fetches binaries from somewhere-outside-mozilla?

Comment 8

[Rob Thijssen [:grenade (EET/UTC+0300)]]

everything is in tooltool. additionally, there are some undocumented features of occ to support tooltool artefact management that i will now document here. :)

if you add an element with ComponentType: ExeInstall, MsiInstall or ZipInstall, to any json manifest under https://github.com/mozilla-releng/OpenCloudConfig/tree/master/userdata/Manifest, and omit the sha512 signature for the artefact and commit that back to the occ repository, occ ci (in https://github.com/mozilla-releng/OpenCloudConfig/blob/master/ci/update-tooltool-repo.sh) will download the artefact from the 3rd party url and then upload it to tooltool and create a git patch in the ci artefacts that can be used to add the sha512 signature to the manifest.

the presence of the sha512 signature in the manifest, indicates (to the install sequence in occ) that the artefact is available from tooltool.

a manifest snippet for an artefact that will be downloaded from a third party provider:

    {
      "ComponentName": "ProcessExplorer",
      "ComponentType": "ZipInstall",
      "Url": "https://download.sysinternals.com/files/ProcessExplorer.zip",
      "Destination": "C:\\ProcessExplorer"
    }

a manifest snippet for an artefact that will be downloaded from tooltool:

    {
      "ComponentName": "ProcessExplorer",
      "ComponentType": "ZipInstall",
      "Url": "https://download.sysinternals.com/files/ProcessExplorer.zip",
      "Destination": "C:\\ProcessExplorer",
      "sha512": "ef89598b2d7f4659a..."
    }

Comment 9

[Pete Moore [:pmoore][:pete]]

Awesome, thanks Rob!