1198374 - Get MSVC 2015 to work in automation using tooltool

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(no longer active)]

The way that Visual C++ is currently deployed to our build infrastructure is by asking Releng/IT to install it on the builders.  That usually takes a bunch of time because those people are very busy.

On other platforms, we have moved to using tooltool <https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool> to upload files such as toolchains.  This will allow people with LDAP access to upload a zipped up version of MSVC 2015 to tooltool that can be referenced later in a tooltool manifest such as https://dxr.mozilla.org/mozilla-central/source/browser/config/tooltool-manifests/win32/releng.manifest.  This is a much more flexible way to deploy new toolchains for using on the try server and/or the build infrastructure, and I think we should do this instead of the old way of deploying the toolchain to builders.

What this all means is that we don't really need help form someone in Releng/IT to green up a build on the try server.  What we need to do is:

1) Someone who is willing to do the work!
2) Zipping up a MSVC bundle from an installation.  This can be done using the chromium script here <https://code.google.com/p/chromium/codesearch#chromium/tools/depot_tools/win_toolchain/package_from_installed.py&sq=package:chromium> from an existing installation.
3) Uploading the zip package to tooltool.
4) Writing a setup.sh script that unpacks the above archive under <mozilla-source-tree>/msvc and adjusts the environment variables set through mozilla-build to point to the extracted Visual C++ installation under <mozilla-source-tree>/msvc.
5) Uploading the above script to tooltool as well.
6) Add the tooltool uploaded files in steps 3 and 5 to the manifest and push to try and get the build to work.

This process should be fairly straightforward.

Comment 1

[Xidorn Quan [:xidorn] UTC+10]

There is one issue which frequently appears when I build Firefox with VS2015 that, it raises an internal compiler error when compiling the js part. It could be an realy annoying intermittent failure when we add VS2015 to our infra.

Comment 2

[(no longer active)]

(In reply to Xidorn Quan [:xidorn] (UTC+12) from comment #1)
> There is one issue which frequently appears when I build Firefox with VS2015
> that, it raises an internal compiler error when compiling the js part. It
> could be an realy annoying intermittent failure when we add VS2015 to our
> infra.

Can you please file a bug about that with some details?

Comment 3

[Xidorn Quan [:xidorn] UTC+10]

Filed bug 1205225.

Comment 4

[sajitk]

Hi, I have a bundle prepared for tooltool, but I guess I don't have appropriate permissions to upload. The message I get is: 
  ERROR - Forbidden: no permission to upload public files

Can you please let me know how I can obtain this permission?
Thanks,

Comment 5

[(no longer active)]

You need to file a bug similar to bug 1081342 and ask for access to the vpn_tooltooleditors group.  That bug is how I got my access.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Any binary blob that gets uploaded to tooltool essentially becomes part of the Gecko source code by virtue of its sha512 appearing in-tree.  You can think of it as a weird form of Github's LFS.

That means a few things:

 * The process of building the blob to be uploaded needs to be clearly documented, preferably in-tree, and preferably as a script.  That's probably some version of the script in #2?

 * The person who reviews the manifest change (adding/updating the sha512) needs to have a high degree of confidence about what is in the blob, either by downloading and analyzing it, or by generating it directly.

 * Copyright and licensing have to be attended to.  I don't know the details with visual studio, but I'd be surprised to find that Microsoft was fine with us throwing a copy of it up on the Internet for anyone to download.  There are ways around this (as we've done with Google's Android SDK and NDK), but they are clumsy and make it much more difficult for people who have not signed the Mozilla NDA to participate in development.  If Chromium is distributing this zip file, though, we can either copy their justification (if we find it convincing), or maybe just copy their blob.

As far as uploading for your try jobs, it will be a good bit easier to just upload the binary somewhere like S3 or people.mozilla.org and point to an explicit URL.  Once you're confident that it's working, we can reproduce the binary and add it to tooltool.

As a final note, as we transition Windows builds to TaskCluster, we'll be looking at similar techniques for importing the toolchains.  Most likely that will involve "read-only pre-populated caches", meaning that we run a TaskCluster task once to generate the compiler using an in-tree script, and then the resulting directory structure is magically mounted on every subsequent job.  If you modify the script, you get a new compiler built for you before your build job starts.

Comment 7

[Biru [:poiru]]

I'll give this a shot. I'm thinking of trying to use the new standalone build tools package: http://blogs.msdn.com/b/vcblog/archive/2015/11/02/announcing-visual-c-build-tools-2015-standalone-c-tools-for-build-environments.aspx

Comment 8

[Biru [:poiru]]

With the dependencies fixed (all of which have patches), I was able to build Firefox on Try with a packaged VS2015 (downloaded from my own server, not in tooltool). After the dependcies are resolved, I'll present the packaging script (slightly modified from Chromium's) and upload the package to tooltool.

dustin, regarding copyright/licensing, Google uses the VS packaging internally and asks external folk download and install VS2015 Community. We should probably do the same (and perhaps upload the package to some other MoCo-only location as well so that MoCo employees can get it without tooltool). If I upload the VS package to tooltool with the internal token, will the builders be able to download it?

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Yes, the builders can download internal files (I think - I know Linux already has that set up in Buildbot, but I'm not 100% sure about Windows.  At any rate, not that hard).

I imagine it will add quite a bit to our build times to download and install VS on every build, right?  How long do the download and install steps take?

Bear in mind that tooltool is content-addressed, so unless non-NDA people have exactly the same bits (same sha512 hash), tooltool will refuse to look at their blob.

Proprietary toolchains stink :(

Comment 10

[Biru [:poiru]]

(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> I imagine it will add quite a bit to our build times to download and install
> VS on every build, right?  How long do the download and install steps take?

We're not downloading and installing VS, we're just inflating an archive of a previous VS installation. A .tar.bz2 archive of VS and the Windows SDK is about 350M and contains ~7000 files that consume 1.6GB uncompressed.

We already do something similar on other platforms. For example, on OS X, the clang and rustc packages are ~300MB and take about 45 seconds to unpack. Download time is probably not very important thanks to the tooltool cache. We should probably optimize for fast decompression (instead of high compression).

> Bear in mind that tooltool is content-addressed, so unless non-NDA people
> have exactly the same bits (same sha512 hash), tooltool will refuse to look
> at their blob.

I'm not sure why normal users (NDA or non-NDA) would need to use tooltool at all. They can just install VS normally like they always have. (That said, NDA people would have the option of grabbing our VS toolchain archive if they only need the compilers etc. and not the IDE.)

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Depending on which layer of the build system (mozharness, mach, or make) invokes tooltool, it can be easier or harder to avoid using it when running local builds.  Anyway, sounds like you have this worked out!

Comment 12

[Xidorn Quan [:xidorn] UTC+10]

FWIW, it was reported by :jgilbert on IRC that the code after bug 1230034 doesn't build on VS2015 without Update 1. So we would need VS2015u1 in our infra.

Comment 13

[Xidorn Quan [:xidorn] UTC+10]

Using Update 1 has been required for VS2015 since bug 1244092.

Comment 14

[Gregory Szorc [:gps]]

I've been actively working on this as part of my overall work to switch over automation builds to VS2015u1.

Comment 15

[Gregory Szorc [:gps]]

The scope of this is larger than Try: we're going to attempt to use tooltool for managing the Windows toolchain for the actual builders. This means that we'll be able to upgrade the Windows toolchain without requiring releng to make changes to the builder images.

The risk here is that tooltool won't be fast enough (for example it purges cached files). My preliminary measurements show that downloading a self-contained zip file (script to produce it is in bug 1253707) takes 1-2 minutes. I can likely make that faster with some I/O tricks. With an active tooltool cache, hopefully that delay is only a few seconds.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Is the unpacking itself slow?

Comment 17

[Gregory Szorc [:gps]]

The unpacking appears to take a minute or two. However, compilation with VS2015 appears to be a minute or two faster, so it offsets.

Since tooltool performance is important to us, I can follow up with some performance improvements to tooltool. I know a few tricks (mainly http://gregoryszorc.com/blog/2015/10/22/append-i/o-performance-on-windows/).

Comment 18

[Gregory Szorc [:gps]]

Improving tooltool's uncompress performance is tracked in bug 1245553.

Comment 19

[Gregory Szorc [:gps]]

The latest patch on bug 1186060 has VS2015 installing via tooltool. We just need to figure out specifics of what goes in the archive. And that's bug 1253707. I don't see a need for this bug that isn't covered elsewhere, so I'm calling it complete.