563319 - Test performance of Visual C++ 2010 build vs. current trunk nightly build

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[(not currently active) Ted Mielczarek]

I'd like to switch the trunk nightly builds to VC++ 2010 (bug 563318), since it will fix some bugs with symbol dumping. I think it may also improve performance slightly, but that bears investigation. We should probably test some builds just to verify that it doesn't negatively impact performance, at least. We'll need to fix bug 563316 first to get an apples-to-apples comparison.

Comment 1

[Matheus Kerschbaum]

First attempt at comparison: http://goo.gl/wgxSE

Ignoring what looks like noise, the differences are:
  - Small win on tp4
  - 10% tp4_memset regression on win7. On a second run this was only a 4.23% regression, but it still worries me that we are regressing at all.
  - Small win on tp5_pbytes_paint
  - Small win across all dromaeo benchmarks
  - Small win on tsspider_* tests
  - Significant (8.7%) win on tspaint on winxp, small regression (noise, most likely) on the same test on win7
  - Significant wins on ts_places_* benchmarks except ts_places_generated_med_shutdown where both platforms regress a bit.

Apart from the weird result on tp4_memset this looks like a minor win on most benchmarks.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Fantastic.  Thanks Matheus!

Comment 3

[(not currently active) Ted Mielczarek]

Thanks for running that comparison!

Comment 4

[(not currently active) Ted Mielczarek]

Interesting. So tp4_memset is the working set of the process, sampled during Tp4. Tp5_memset_paint (the replacement metric), doesn't seem to have budged, so I don't think this is an across-the-board regression.

Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22??? That's crazytown!

https://wiki.mozilla.org/Buildbot/Talos#Working_Set_.28tp5_memset.29

Comment 5

[Justin Lebar (not reading bugmail)]

Looks like you avoided bug 696196 by building the original revision on try yourself.  That's a good idea.

> Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22??? 
> That's crazytown!

When you call it 32mb, it doesn't sound so ginormous.  :)  Anyway, that test looks pretty noisy.

http://graphs-new.mozilla.org/graph.html#tests=[[45,1,12],[45,1,1]]&sel=none&displayrange=365&datatype=running

Matheus, I might kick more runs of these tests, since there's a large amount of noise on some of them.  Then you'll be able to better quantify the various small wins.

If you really want to be sure, you may also want to kick a few new *builds*, just in case there's some randomness in the performance of the builds.  Good luck using compare-talos there.  :)

Comment 6

[Matheus Kerschbaum]

New builds with new results: http://goo.gl/lXKpG

Comparing with the first set of results:
  - The tp4 win is definitely real.
  - There might be a very minor tdhtml_paint regression.
  - The 10% regression on tp4_memset was probably noise, but tp4_memset does regress a little.
  - tp4_shutdown had some weird results on this run (21% regression on winxp and a 14% win on win7).
  - tp5_paint might be a minor win too.
  - The tp5_pbytes_paint result was probably noise.
  - On this run tp5_shutdown_paint was a significant win (8-13%).
  - The dromaeo benchmarks are quite noisy but dromaeo_dom is definitely a win.
  - The tsspider_* tests show a consistent win.
  - tspaint is noisy on win7 but the ~9% win on winxp is surprisingly consistent.
  - The ts_places_* tests are quite noisy as well but show a overall win.

Overall only tdhtml_paint and tp4_memset show a consistent regression with everything else either staying the same or improving a little.