1080624 - win8x64 talos needs work before turning it on

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently we don't run any talos tests on win8x64.  We currently run talos tests on win8 machines but 32 bit versions of firefox.

if we switch 100% on v.36 branches from 32 to 64 bit, we *could* keep the same graphs and ignore any regressions when v.36 uplifts to aurora.  Otherwise we will need to:
* add a flag to talos to indicate we are runing 64 bit vs default 32, and then modify the machine name so we post results to a 64 bit *platform*
* invoke talos with this flag via the mozharness script
* fix try chooser to support this
* modify graph server to have duplicate definitions of all win8 machines (duplicate it with a .64 or something)
* deploy graph server fix, talos fix, mozharness fix


We still need to test that win64 builds will actually run talos!

do we want to live with ignoring win8* regressions on uplift?
do we want to live with an ugly hack to talos and graph server to support reuse of the same machines?
who is going to enable talos tests on these platforms to verify the tests run?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

bsmedberg, please comment on the questions in comment 0.

keep in mind that this doesn't fully outline all the work, there always seems to be unknowns.

Comment 2

[Benjamin Smedberg]

For unit tests, we're planning on dropping win8-win32 and running win8-win64. I assume we don't have the hardware capacity to run both win8-win32 and win8-win64 talos tests always. Has there been a history of win8 Talos regression on win8 that don't show up on win7?

#1 priority is having automated perf tests with a new baseline.

#2 priority is to have some comparisons of win32 and win64 on identical hardware. I think this could be a manual comparison. It will help us understand how hard to push win64 as a perf improvement to the product.

Comment 3

[Armen [:armenzg]]

We have Firefox 64-bit talos jobs running on Date.
https://tbpl.mozilla.org/?tree=Date&jobname=win8_64%20date%20talos

We now need to compare how they fair compared to Fx 32-bit builds.

Comment 4

[Armen [:armenzg]]

I've pushed again to Cedar since we had some 32-bit talos results for Win8 on our last push since we had not yet transitioned.
https://tbpl.mozilla.org/?tree=Date&jobname=win8_64%20date%20talos&rev=de178d3151a2

After that we can run more talos jobs on it and compare it to Fx 32-bit on m-c.

Comment 5

[Armen [:armenzg]]

The results are back and no 32-bit builds were run on Win8:
https://tbpl.mozilla.org/?tree=Date&jobname=talos&rev=de178d3151a2

I will look into creating extra jobs and compare it with m-c.

Comment 6

[Armen [:armenzg]]

I modified a script to allow me to trigger a lot of talos jobs on Date.
We should have more data points tomorrow.

https://tbpl.mozilla.org/?tree=Date&jobname=win8_64%20date%20talos

Comment 7

[Armen [:armenzg]]

Attachment: switch win8 32-bit talos for win8 64-bit on trunk trees

This patch is for the future and not to be landed until we're ready to switch, however, I can't figure out why it does not add win8_64 talos jobs.

What am I missing?

Comment 8

[Jonathan Griffin (:jgriffin)]

This is probably because of this block:

http://hg.mozilla.org/build/buildbot-configs/file/a18ca2565e6d/mozilla-tests/config.py#l2019

Comment 9

[Armen [:armenzg]]

Attachment: switch win8 32-bit talos for win8 64-bit on trunk trees

jgriffin good catch! Thanks.

This patch does not touch unittest builders. It only swaps Fx 32-bit for Fx 64-bit on talos for Windows 8 machines.

I'm leaving the patch in here until we determine when we want to switch one for the other.

Comment 10

[Armen [:armenzg]]

Passing to jmaher to get the comparison.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

comparison is all good- from a performance regression detection point of view, we are good to go in switching from 32 bit to 64 bit on all trees we will switch the unittests.

Comment 12

[Benjamin Smedberg]

\o/

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

I am going to resolve this and sort out the issue where in live production all results are posted to the PGO branch instead of non-pgo.