578767 - Comparison runs of 32-bit and 64-bit Windows builds on the same machine

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Benjamin Smedberg]

In order to know whether 64-bit builds are faster than 32-bit builds on Windows, we need to run them on the same OS (currently 32-bit builds are tested on 32-bit windows). This is a request to do a set of talos runs of equivalent 32-bit and 64-bit nightlies to the point where we can have some statistical confidence in the results numbers (the current 64-bit Talos numbers seem to have a lot of variance).

Comment 1

[[e-mail] (:bhearsum)]

I can do this

Comment 2

[[e-mail] (:bhearsum)]

I did multiple test runs of Dromaeo and Sunspider on a 64-bit test machine today. Quick results:
* There were 19 tests in which the 32-bit build beat the 64-bit one
* There were 26 tests in which the 64-bit build beat the 32-bit noe
* There were 30 tests where the numbers were very very close or the test never completed on one or both builds. 

The most dramatic wins for the 64-bit build were
* Dromaeo's Code Evaluation test - 50% faster
* Sunspider's 3d morph test - 37% faster

The most dramatic losses for the 64-bit were:
* Dromaeo's 3D Mesh Transform test - 124% slower
* Sunspider string base64 test - 108% slower
* Dromaeo's Bitwise And test - 80% slower
* Dromaeo's Dom Query test - 75% slower

Full results are here: http://spreadsheets.google.com/ccc?key=0AhwotqoBm6lEdHp3LXNmQ0l1eFpabjJYVFRTdndGYUE&hl=en#gid=0

Comment 3

[[e-mail] (:bhearsum)]

I'm not planning on doing any more tests unless someone requests them. Please re-open if you want additional data.

Comment 4

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

http://groups.google.com/group/mozilla.dev.planning/msg/f5c73ec86a77c032 notes this isn't an apples to apples comparison, though, because the 64-bit builds:
 * are with MSVC10 rather than MSVC9
 * do not have jemalloc
 * do not have PGO

Comment 5

[[e-mail] (:bhearsum)]

Yeah, it's a good point. I can re-run once the 64-bit builds are comparable.

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to comment #4)
> http://groups.google.com/group/mozilla.dev.planning/msg/f5c73ec86a77c032 notes
> this isn't an apples to apples comparison, though, because the 64-bit builds:
>  * are with MSVC10 rather than MSVC9

32-bit builds are still with MSVC8, FYI.

Comment 7

[[e-mail] (:bhearsum)]

Hah, that's funny. Well, unless we're planning to bump the compiler for 32-bit builds I'd argue that MSVC9 64-bit vs MSVC8 32-bit is the most accurate comparison.

Comment 8

[(not currently active) Ted Mielczarek]

Yeah, that seems fair. jemalloc + PGO is a pretty big difference though, so we should fix those before re-measuring.

Comment 9

[[e-mail] (:bhearsum)]

We're not shipping a 64-bit Windows Firefox 4, reso incomplete for now.