Last Comment Bug 563319 - Test performance of Visual C++ 2010 build vs. current trunk nightly build
: Test performance of Visual C++ 2010 build vs. current trunk nightly build
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86 Windows 7
-- normal (vote)
: ---
Assigned To: Matheus Kerschbaum
: Gregory Szorc [:gps] (away until 2017-03-20)
Depends on: 515492
Blocks: msvc2010
  Show dependency treegraph
Reported: 2010-05-03 05:35 PDT by Ted Mielczarek [:ted.mielczarek]
Modified: 2012-01-31 12:45 PST (History)
19 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Description User image Ted Mielczarek [:ted.mielczarek] 2010-05-03 05:35:57 PDT
I'd like to switch the trunk nightly builds to VC++ 2010 (bug 563318), since it will fix some bugs with symbol dumping. I think it may also improve performance slightly, but that bears investigation. We should probably test some builds just to verify that it doesn't negatively impact performance, at least. We'll need to fix bug 563316 first to get an apples-to-apples comparison.
Comment 1 User image Matheus Kerschbaum 2011-11-15 16:44:14 PST
First attempt at comparison:

Ignoring what looks like noise, the differences are:
  - Small win on tp4
  - 10% tp4_memset regression on win7. On a second run this was only a 4.23% regression, but it still worries me that we are regressing at all.
  - Small win on tp5_pbytes_paint
  - Small win across all dromaeo benchmarks
  - Small win on tsspider_* tests
  - Significant (8.7%) win on tspaint on winxp, small regression (noise, most likely) on the same test on win7
  - Significant wins on ts_places_* benchmarks except ts_places_generated_med_shutdown where both platforms regress a bit.

Apart from the weird result on tp4_memset this looks like a minor win on most benchmarks.
Comment 2 User image Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2011-11-16 05:37:10 PST
Fantastic.  Thanks Matheus!
Comment 3 User image Ted Mielczarek [:ted.mielczarek] 2011-11-16 06:34:36 PST
Thanks for running that comparison!
Comment 4 User image Ted Mielczarek [:ted.mielczarek] 2011-11-16 06:38:22 PST
Interesting. So tp4_memset is the working set of the process, sampled during Tp4. Tp5_memset_paint (the replacement metric), doesn't seem to have budged, so I don't think this is an across-the-board regression.

Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22??? That's crazytown!
Comment 5 User image Justin Lebar (not reading bugmail) 2011-11-16 08:23:22 PST
Looks like you avoided bug 696196 by building the original revision on try yourself.  That's a good idea.

> Also, what's up with the *gigantic* variation number on the new tp4_memset run? ±33912841.22??? 
> That's crazytown!

When you call it 32mb, it doesn't sound so ginormous.  :)  Anyway, that test looks pretty noisy.[[45,1,12],[45,1,1]]&sel=none&displayrange=365&datatype=running

Matheus, I might kick more runs of these tests, since there's a large amount of noise on some of them.  Then you'll be able to better quantify the various small wins.

If you really want to be sure, you may also want to kick a few new *builds*, just in case there's some randomness in the performance of the builds.  Good luck using compare-talos there.  :)
Comment 6 User image Matheus Kerschbaum 2011-11-16 23:18:00 PST
New builds with new results:

Comparing with the first set of results:
  - The tp4 win is definitely real.
  - There might be a very minor tdhtml_paint regression.
  - The 10% regression on tp4_memset was probably noise, but tp4_memset does regress a little.
  - tp4_shutdown had some weird results on this run (21% regression on winxp and a 14% win on win7).
  - tp5_paint might be a minor win too.
  - The tp5_pbytes_paint result was probably noise.
  - On this run tp5_shutdown_paint was a significant win (8-13%).
  - The dromaeo benchmarks are quite noisy but dromaeo_dom is definitely a win.
  - The tsspider_* tests show a consistent win.
  - tspaint is noisy on win7 but the ~9% win on winxp is surprisingly consistent.
  - The ts_places_* tests are quite noisy as well but show a overall win.

Overall only tdhtml_paint and tp4_memset show a consistent regression with everything else either staying the same or improving a little.

Note You need to log in before you can comment on or make changes to this bug.