1356654 - Investigate performance from VS2017's new optimizer

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[(Away)]

Visual Studio 2017 claims to have a new code optimizer: https://blogs.msdn.microsoft.com/vcblog/2016/05/04/new-code-optimizer/

We should compare the perf against VS2015 and decide if it's worth switching compilers.

Comment 1

[Gregory Szorc [:gps]]

VS2017's code targets the same as VS2015. So my opinion is we should switch to VS2017 at our earliest convenience provided there are no major regressions. A big benefit of switching is we can force people to VS2017 on the IDE side, which has much better performance on large projects.

Comment 2

[(no longer active)]

This was done, and there was no advantage in Speedometer V2 with the new optimizer enabled.

Comment 3

[Guilherme Lima]

Is still possible to make the comparison in another test?
https://bugzilla.mozilla.org/show_bug.cgi?id=1245487#c7 is the only place I remember reading that MSVC was slowing down the JS engine (Octane-CodeLoad affected by inlining heuristics).

Comment 4

[(no longer active)]

(In reply to Guilherme Lima from comment #3)
> Is still possible to make the comparison in another test?

Not sure what has changed for us to expect a different outcome?  Usually it's pointless to try again before the next version of Visual Studio comes out.

> https://bugzilla.mozilla.org/show_bug.cgi?id=1245487#c7 is the only place I
> remember reading that MSVC was slowing down the JS engine (Octane-CodeLoad
> affected by inlining heuristics).

Sorry, I'm not sure if I follow that part.  But at any rate, we don't actively measure against Octane any more.  The performance difference of MSVC 2017 vs 2015 on Firefox was measured on Speedometer V2, where we observed no measurable difference.

Comment 5

[(Away)]

> The performance difference of
> MSVC 2017 vs 2015 on Firefox was measured on Speedometer V2, where we
> observed no measurable difference.

To clarify: VS2017's SSA optimizer was also shipped in VS2015 Update 3, so in theory we're already using it as of bug 1283203. But we weren't sure if it was enabled by default, so we measured VS2015 with SSA explicitly enabled versus VS2015 with SSA explicitly disabled, and there was no difference on Speedometer.

Comment 6

[(Away)]

Our previous measurements were a bit oversimplified. We looked at "VS2015u3 with SSA on" versus "VS2015u3 with SSA off" on the assumption that the new SSA engine was the only performance improvement offered by VS2017.

According to this presentation: https://youtu.be/jsdn3kXFVdA?t=13m12s, VS2017 has performance improvements above and beyond the SSA optimizer that was backported to VS2015u3.

So I decided to do a more proper comparison between a VS2015u3 PGO build (SSA on-by-default, this should match what we're building with officially) and a real VS2017 PGO build.

Speedometer v2 average of 7 runs, x64:
VS2015: 144.0
VS2017: 152.3 (+5.76%)

Ehsan: what do you think about reopening this bug for post-57?

Comment 7

[(no longer active)]

Your reasoning sounds solid to me, thanks!

Comment 8

[(Away)]

I'm going to say that this "investigate" task is complete, with the result being that we switched to VS2017 in bug 1408789.