Closed Bug 1186060 (vs2015) Opened 9 years ago Closed 8 years ago

Use Visual Studio 2015 Update 1 for release builds

Categories

(Firefox Build System :: General, defect)

defect
Not set
normal

Tracking

(firefox42 affected, firefox48 fixed)

RESOLVED FIXED
mozilla48
Tracking Status
firefox42 --- affected
firefox48 --- fixed

People

(Reporter: briansmith, Assigned: gps)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

Attachments

(1 file)

Let's make bug 1119082 be about getting Visual Studio 2015 support to the point where we support Visual Studio 2015 as a tier 2 type platform, where we try to get bugs that break Visual Studio 2015 builds fixed ASAP.

Let's have *this* bug track making Visual Studio 2015 the compiler for actual releases.
No longer blocks: 1186070
What's the primary motivation here? Updating toolchains is a fair bit of work, it's good to know why we're doing it.
The primary motivation is to facilitate bug 1186064 to make more C++ language and library features available in Gecko. See http://blogs.msdn.com/b/vcblog/archive/2015/06/19/c-11-14-17-features-in-vs-2015-rtm.aspx about the new features in VS2015. This will allow more Mozilla-isms to be removed and make code easier to write.
I second that.  We absolutely should not let 2013 become the next 2010.  I would really like to see us switch to 2015 this year, and also do it through tooltool instead of waiting for ages for the new Visual Studio install to be deployed to all of our build machine, which is a huge part of why updating toolchains is difficult.
On a separate note, there's something I've never seen talked about, but with the rate at which we're upgrading, maybe that should weigh in: the cost of upgrade for third parties. I'm saying this without knowing how much it costs to upgrade from 2013 to 2015, nor how much it cost to upgrade from 2010 to 2013, but for third parties that do non-open-source derivative work (addons for Firefox, apps based on Gecko, etc. ; those are not hypothetical, there are plenty of them), a bump in the minimum version we require is a hit in the wallet. And we already forced 2013 only 6 months ago.

Not saying this is a blocker and we shouldn't bump to 2015, but that those things should be part of the conversation as well, especially considering the timeline.
(In reply to Mike Hommey [:glandium] from comment #4)

Isn't the concern about wallet impact nullified by Visual Studio Community Edition, which is free for open source? IMO Community Edition is a huge reason to upgrade because everyone will be able to use a free and powerful edition instead of a) paying b) using the feature-pruned Express edition.

I do hear the concern about upgrading too frequently. Visual Studio installations are massive and not a lot of fun to deal with.
(In reply to Gregory Szorc [:gps] from comment #5)
> (In reply to Mike Hommey [:glandium] from comment #4)
> 
> Isn't the concern about wallet impact nullified by Visual Studio Community
> Edition, which is free for open source?

You missed the part where I said non-open-source. To cite one that I'm aware of because they're having problems at the moment (unrelated to MSVC version): Symantec's Norton toolbar is one example.
MSVC has had a free edition (Express) for years which can be used for open source or closed source development.  The only thing that the Community Edition gives you on top of that is the IDE, and since side by side installations of Visual Studio works perfectly fine, you could install VS2015 Express and keep your old 2010/etc edition for debugging and other IDE related stuff.

I don't think we are forcing a monetary cost on these consumers.
Depends on: 1200336
Any plans to move this forward? This seems pretty important to me.
Nobody has given a super-compelling reason to do it, it's a lot of work, and the dependent bugs aren't even all fixed, so I'm not sure it's even feasible yet.
With bug 1124017 fixed, a VS2015 build of Firefox runs perfectly fine in XP SP2 and SP3!

AFAICT, we still need to fix CRT DLL packaging (bug 1204202) and get VS2015 toolchain on the build machines (bug 1186060 to start).
FWIW, Chromium appears to have switched to VS2015 ~2 months ago (https://codereview.chromium.org/1514853004/). A PGO bug fix was cited in https://codereview.chromium.org/1514853004/#msg11.
Preliminary measurements in bug 1250797 indicate that VS2015 PGO builds are ~1.4x faster than VS2013.
Per bug 1250797 comment 12, PGO builds on VS2015 are ~1.8x than VS2013. The speedup increased over comment 12 because the 2nd PGO link phase is nearly 3x faster on VS2015 (the instrumentation phase is only ~2x faster).

I haven't yet tested the speedup in automation. But surely there will be a speedup.
Depends on: 1250797
Depends on: 1252556
Depends on: 1252589
I've been assigned to shepherd this from the build system perspective.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Summary: Use Visual Studio 2015 for release builds → Use Visual Studio 2015 Update 1 for release builds
Depends on: 1198374
Depends on: 1254017
I appear to have 64-bit builds mostly working (minus the crash reporter): https://treeherder.mozilla.org/#/jobs?repo=try&revision=343a8e570f7a

I'm optimistic a PGO try run at https://treeherder.mozilla.org/#/jobs?repo=try&revision=95a24d38f004 will complete in 1-2 hours.

Bug 1242005 is preventing 32-bit builds from completing due to a compiler warning being promoted to an error :(

Hopefully we can start looking for Talos and other test regressions in the next ~24 hours...
(In reply to Gregory Szorc [:gps] from comment #25)
> I appear to have 64-bit builds mostly working (minus the crash reporter):
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=343a8e570f7a

The latest patch in bug 623183 should fix the crash reporter bit.
Depends on: 1254638
Depends on: 1254767
Depends on: 1254779
(THIS PATCH IS NOT READY FOR FINAL REVIEW BECAUSE OF THE HACKINESS
AROUND INSTALLING THE TOOLCHAIN. I'm submitting the patch now
to test the waters around the approach.)

This commit switches Windows builds from Visual Studio 2013
to Visual Studio 2015 Update 1.

Previously, Visual Studio was installed on the builers as part
of the base system image. Starting with this commit, we obtain
Visual Studio from a pre-generated, self-contained archive
containing the executables, Windows SDK, and other support
files. This means that new Windows toolchains can be installed
without having to modify configuration of machines in automation!

The mozconfigs for Visual Studio 2015 are a bit different from
existing mozconfigs.

Because it appears to be completely redundant and not necessary,
the LIBPATH variable has been dropped.

The order of paths in PATH, LIB, and INCLUDE has changed. The new
order more accurately reflects what would be defined by
vcvarsall.bat.

As part of switching to Visual Studio 2015, the Universal CRT is
now required. So, the 2015 mozconfigs export WIN_UCRT_REDIST_DIR
to define the location to those files.

The switch to Visual Studio 2015 also involves the switch from
the Windows 8.1 SDK to the Windows 10 SDK. However, we still
target an old version of Windows, so this hopefully shouldn't
have any significant fallout.

It's worth noting that switching to Visual Studio 2015 makes
builds - especially PGO builds - significantly faster. Our
PGO build times in automation are ~1 hour faster. Our regular
builds appear to be a few minutes faster.

Review commit: https://reviewboard.mozilla.org/r/38833/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/38833/
Attachment #8728174 - Flags: review?(ted)
Attachment #8728174 - Flags: review?(ehsan)
https://reviewboard.mozilla.org/r/38833/#review35499

::: python/mozbuild/mozbuild/mach_commands.py:324
(Diff revision 1)
> +        # Hacky Windows only VS2015 toolchain application.
> +        from cStringIO import StringIO
> +        import shutil
> +        import requests
> +        import zipfile
> +
> +        msvc_dir = os.path.join(self.topsrcdir, 'vstoolchain')
> +        if os.path.exists(msvc_dir):
> +            shutil.rmtree(msvc_dir)
> +
> +        os.mkdir(msvc_dir)
> +        print('downloading Visual Studio 2015 toolchain...')
> +        toolchain = StringIO(requests.get('https://s3-us-west-2.amazonaws.com/moz-packages/temp/vs2015-3.zip').content)
> +        toolchain.seek(0)
> +        print('uncompressing')
> +        with zipfile.ZipFile(toolchain, mode='r') as zf:
> +            zf.extractall(msvc_dir)
> +        print('finished installing Visual Studio 2015 toolchain')

WTF is this?
https://reviewboard.mozilla.org/r/38833/#review35499

> WTF is this?

The reason for the UPPERCASE comment in the commit message. The plan is to use tooltool in the final version.
Comment on attachment 8728174 [details]
MozReview Request: Bug 1186060 - Build with Visual Studio 2015 Update 1; r?ted, ehsan

https://reviewboard.mozilla.org/r/38833/#review35681

Looks good to me, sans the changes in mach_commands.py which I expect will be removed when this package is put into tooltool.

Out of curiosity, why haven't you put this into tooltool yet?
Attachment #8728174 - Flags: review?(ehsan) → review+
Depends on: 1255656
Comment on attachment 8728174 [details]
MozReview Request: Bug 1186060 - Build with Visual Studio 2015 Update 1; r?ted, ehsan

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/38833/diff/1-2/
https://reviewboard.mozilla.org/r/38833/#review35681

The latest patch has things in tooltool (as an internal file).

I was waiting to put things in tooltool because I didn't want to upload a few GB of archives I knew would likely have to be thrown away.
Depends on: 1255686
Alias: vs2015
Depends on: 1253707
Comment on attachment 8728174 [details]
MozReview Request: Bug 1186060 - Build with Visual Studio 2015 Update 1; r?ted, ehsan

https://reviewboard.mozilla.org/r/38833/#review36193

::: browser/config/mozconfigs/win32/l10n-mozconfig:14
(Diff revision 2)
>  export MOZILLA_OFFICIAL=1
>  
>  if test "$PROCESSOR_ARCHITECTURE" = "AMD64" -o "$PROCESSOR_ARCHITEW6432" = "AMD64"; then
> -  . $topsrcdir/build/win32/mozconfig.vs2013-win64
> +  . $topsrcdir/build/win32/mozconfig.vs2015-win64
>  else
>    . $topsrcdir/build/win32/mozconfig.vs2010

Do we have anything that's still running on 32-bit Windows machines that needs this? (You don't necessarily have to fix this now.)

::: browser/config/tooltool-manifests/win32/releng.manifest:23
(Diff revision 2)
>  "algorithm": "sha512",
>  "filename": "sccache.tar.bz2",
>  "unpack": true
> +},
> +{
> +"size": 441571009,

Per glandium's other bug, it would be nice to have a version annotation here indicating exactly what version of VC2015 this is.

::: build/win32/mozconfig.vs2015-win64:13
(Diff revision 2)
> +export PATH="${vspath}/VC/redist/x86/Microsoft.VC140.CRT:${vspath}/VC/redist/x64/Microsoft.VC140.CRT:${vspath}/SDK/Redist/ucrt/DLLs/x86:${vspath}/SDK/Redist/ucrt/DLLs/x64:${PATH}"
> +
> +export INCLUDE="${vspath}/VC/include:${vspath}/VC/atlmfc/include:${vspath}/SDK/Include/ucrt:${vspath}/SDK/Include/shared:${vspath}/SDK/Include/um:${vspath}/SDK/Include/winrt:${vspath}/DIASDK/include"
> +export LIB="${vspath}/VC/lib:${vspath}/VC/atlmfc/lib:${vspath}/SDK/lib/ucrt/x86:${vspath}/SDK/lib/um/x86:${vspath}/DIASDK/lib"
> +
> +. $topsrcdir/build/mozconfig.vs-common

Do we still need this?
https://dxr.mozilla.org/mozilla-central/source/build/mozconfig.vs-common

::: build/win64/mozconfig.vs2015:1
(Diff revision 2)
> +vspath="$(cd ${topsrcdir} && pwd)/vs2015u1"

These two mozconfigs look similar enough that it feels like you could make them shared with a parameter or two, but I'll leave it up to your judgement whether you think that's warranted.

::: js/src/devtools/automation/winbuildenv.sh:1
(Diff revision 2)
>  # We will be sourcing mozconfig files, which end up calling mk_add_options with

I have...never seen this file before. I assume that since you've got things going on Try you'd notice if you actually broke this!
Attachment #8728174 - Flags: review?(ted) → review+
https://reviewboard.mozilla.org/r/38833/#review36193

> I have...never seen this file before. I assume that since you've got things going on Try you'd notice if you actually broke this!

Well, I don't have SM working in automation yet. See bug 1255686. That's one of the major open issues.
Blocks: 1256997
As of yesterday, some SM builds are working. However, some are failing because vcruntime140d.dll isn't on PATH. And it isn't in the zip archive either, so that will need to be updated.

Also, the Try spam is to get bug 1124033 sorted out.
According to glandium SM builds are apparently configured incorrectly and shouldn't be using the debug runtime. Will file another bug.
Depends on: 1257005
Depends on: 1257435
Depends on: 1257722
https://reviewboard.mozilla.org/r/38833/#review36193

> Do we still need this?
> https://dxr.mozilla.org/mozilla-central/source/build/mozconfig.vs-common

I'm, uh, not sure. Feels like a follow-up bug to me.

> These two mozconfigs look similar enough that it feels like you could make them shared with a parameter or two, but I'll leave it up to your judgement whether you think that's warranted.

Yes, they are pretty similar. However, I'm leaning towards holding off on this. I'd rather do the tidying as a follow-up, since what this patch implements conforms to the existing "style" of mozconfigs more or less.
Comment on attachment 8728174 [details]
MozReview Request: Bug 1186060 - Build with Visual Studio 2015 Update 1; r?ted, ehsan

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/38833/diff/2-3/
No longer depends on: 1252556
Comment on attachment 8728174 [details]
MozReview Request: Bug 1186060 - Build with Visual Studio 2015 Update 1; r?ted, ehsan

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/38833/diff/3-4/
Blocks: 1259488
Blocks: 1259499
https://reviewboard.mozilla.org/r/38833/#review36193

> I'm, uh, not sure. Feels like a follow-up bug to me.

Can you file a followup on this?
For posterity, the zip file containing the Visual Studio 2015 and SDK files was produced in the following manner:

1) I installed Windows 7 Ultimate 64-bit into a Hyper-V VM running on my personal desktop at home from a physical DVD that was purchased from Microsoft
2) All required updates through Windows Update were installed. This required multiple reboots. This process started a few minutes after finishing the OS install and finished the morning after (I slept in the middle with the host OS locked).
3) The instructions at https://hg.mozilla.org/mozilla-central/file/8ae2e73a7d90/build/docs/toolchains.rst#l43 were followed to install VS2015u1 Community and produce a zip archive.
4) I copied the zip file to a separate Linux VM to upload it to the internal tooltool (I verified the SHA-512 to ensure the file was identical across copying)

For the tin foil hat crowd, yes, there is a single point of trust and you have to trust that I didn't act maliciously or that my home machine / VM wasn't compromised. I would encourage people to reproduce my steps and compare against what I uploaded to tooltool. I'm actually quite interested in seeing if someone else can get a bit-for-bit identical zip archive. In theory it is possible. But so far only I have produced the zip archives. I would love for someday this process to be performed in our automation. I'm guessing this is blocked on TaskCluster support for Windows.
Blocks: 982928
Blocks: 1259532
https://hg.mozilla.org/mozilla-central/rev/b942c98f56c4
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Blocks: 1259782
Depends on: 1262567
Depends on: 1262731
Blocks: 1264499
Blocks: 1265460
Blocks: 1265615
Blocks: 1035125
Depends on: 1270664
Depends on: 1270714
Depends on: 1320738
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.