1092428 - [e10s] Run Talos on e10s builds (at a reduced frequency)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris Peterson [:cpeterson]]

In our recent meeting [1] about e10s testing, we concluded that:

1. Before e10s is enabled by default, we will run Talos tests on PGO, non-PGO, Linux e10s, and Windows e10s+PGO. To reduce test load, jmaher suggests we only test e10s on Win7, Win8, 32-bit Linux, and 64-bit Linux.

2. After e10s is enabled by default in Nightly, will we test e10s and non-e10s on every push?

3. After e10s is riding the trains (Aurora), we will test e10s on every push but non-e10s on the reduced testing frequency (above).


[1] https://etherpad.mozilla.org/uM2IdRZFJf

Comment 1

[Chris AtLee [:catlee]]

Ok, working on enabling e10s talos suites for PGO builds only to start with.

Comment 2

[Chris AtLee [:catlee]]

Attachment: pgo_talos-buildbotcustom.diff

Comment 3

[Chris AtLee [:catlee]]

Attachment: pgo_talos-buildbot-configs.diff

disable e10s talos suites on holly; enable e10s suites on m-c for pgo builds.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8526084 [details] [diff] [review]
pgo_talos-buildbot-configs.diff

Review of attachment 8526084 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1738,5 @@
>  BRANCHES['mozilla-central']['build_branch'] = "1.9.2"
>  BRANCHES['mozilla-central']['pgo_strategy'] = 'periodic'
> +BRANCHES['mozilla-central']['xperf-e10s_tests'] = (1, False, TALOS_TP_NEW_OPTS, WIN7_ONLY)
> +BRANCHES['mozilla-central']['tp5o-e10s_tests'] = (1, False, TALOS_TP_NEW_OPTS, NO_WINXP)
> +BRANCHES['mozilla-central']['g1-e10s_tests'] = (1, False, TALOS_TP_NEW_OPTS, NO_WINXP)

I suspect this NO_WINXP could be adjusted in the future to match non e10s (ALL_TALOS_PLATFORMS)- there could be a reason for not doing this on xp which I am not aware of right now.

@@ +1750,5 @@
> +                                                  'other-e10s_nol64',
> +                                                  'other-e10s_l64',
> +                                                  'svgr-e10s',
> +                                                  'dromaeojs-e10s',
> +                                                  'chromez-e10s']

will this remove the non e10s stuff from pgo?  In many ways that wouldn't be a big deal;

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 8526084 [details] [diff] [review]
pgo_talos-buildbot-configs.diff

Review of attachment 8526084 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +1750,5 @@
> +                                                  'other-e10s_nol64',
> +                                                  'other-e10s_l64',
> +                                                  'svgr-e10s',
> +                                                  'dromaeojs-e10s',
> +                                                  'chromez-e10s']

No, but maybe that means we should change the name of this configuration key. What this means is that these suites will be run only for PGO builds, not for regular opt builds.

The list of builders added/removed by these changes:
--- old.txt	2014-11-20 11:10:31.000000000 -0500
+++ new.txt	2014-11-20 11:26:13.000000000 -0500
+mozilla-central Windows 8 64-bit mozilla-central pgo talos other-e10s_nol64 ScriptFactory
+mozilla-central Windows 8 64-bit mozilla-central pgo talos svgr-e10s ScriptFactory
+mozilla-central Windows 8 64-bit mozilla-central pgo talos dromaeojs-e10s ScriptFactory
+mozilla-central Windows 8 64-bit mozilla-central pgo talos chromez-e10s ScriptFactory
+mozilla-central Windows 8 64-bit mozilla-central pgo talos tp5o-e10s ScriptFactory
+mozilla-central Windows 8 64-bit mozilla-central pgo talos g1-e10s ScriptFactory
+mozilla-central Windows 7 32-bit mozilla-central pgo talos other-e10s_nol64 ScriptFactory
+mozilla-central Windows 7 32-bit mozilla-central pgo talos xperf-e10s ScriptFactory
+mozilla-central Windows 7 32-bit mozilla-central pgo talos svgr-e10s ScriptFactory
+mozilla-central Windows 7 32-bit mozilla-central pgo talos dromaeojs-e10s ScriptFactory
+mozilla-central Windows 7 32-bit mozilla-central pgo talos chromez-e10s ScriptFactory
+mozilla-central Windows 7 32-bit mozilla-central pgo talos tp5o-e10s ScriptFactory
+mozilla-central Windows 7 32-bit mozilla-central pgo talos g1-e10s ScriptFactory
+mozilla-central Windows XP 32-bit mozilla-central pgo talos other-e10s_nol64 ScriptFactory
+mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos other-e10s_l64 ScriptFactory
+mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos svgr-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos dromaeojs-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos chromez-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos tp5o-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 x64 mozilla-central pgo talos g1-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos other-e10s_nol64 ScriptFactory
+mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos svgr-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos dromaeojs-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos chromez-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos tp5o-e10s ScriptFactory
+mozilla-central Ubuntu HW 12.04 mozilla-central pgo talos g1-e10s ScriptFactory
-holly Windows 8 64-bit holly talos other-e10s_nol64 ScriptFactory
-holly Windows 8 64-bit holly talos svgr-e10s ScriptFactory
-holly Windows 8 64-bit holly talos dromaeojs-e10s ScriptFactory
-holly Windows 8 64-bit holly talos chromez-e10s ScriptFactory
-holly Windows 8 64-bit holly talos tp5o-e10s ScriptFactory
-holly Windows 8 64-bit holly talos g1-e10s ScriptFactory
-holly Rev5 MacOSX Mountain Lion 10.8 holly talos other-e10s_nol64 ScriptFactory
-holly Rev5 MacOSX Mountain Lion 10.8 holly talos svgr-e10s ScriptFactory
-holly Rev5 MacOSX Mountain Lion 10.8 holly talos dromaeojs-e10s ScriptFactory
-holly Rev5 MacOSX Mountain Lion 10.8 holly talos chromez-e10s ScriptFactory
-holly Rev5 MacOSX Mountain Lion 10.8 holly talos tp5o-e10s ScriptFactory
-holly Rev5 MacOSX Mountain Lion 10.8 holly talos g1-e10s ScriptFactory
-holly Rev4 MacOSX Snow Leopard 10.6 holly talos other-e10s_nol64 ScriptFactory
-holly Rev4 MacOSX Snow Leopard 10.6 holly talos svgr-e10s ScriptFactory
-holly Rev4 MacOSX Snow Leopard 10.6 holly talos dromaeojs-e10s ScriptFactory
-holly Rev4 MacOSX Snow Leopard 10.6 holly talos chromez-e10s ScriptFactory
-holly Rev4 MacOSX Snow Leopard 10.6 holly talos tp5o-e10s ScriptFactory
-holly Rev4 MacOSX Snow Leopard 10.6 holly talos g1-e10s ScriptFactory
-holly Windows 7 32-bit holly talos other-e10s_nol64 ScriptFactory
-holly Windows 7 32-bit holly talos xperf-e10s ScriptFactory
-holly Windows 7 32-bit holly talos svgr-e10s ScriptFactory
-holly Windows 7 32-bit holly talos dromaeojs-e10s ScriptFactory
-holly Windows 7 32-bit holly talos chromez-e10s ScriptFactory
-holly Windows 7 32-bit holly talos tp5o-e10s ScriptFactory
-holly Windows 7 32-bit holly talos g1-e10s ScriptFactory
-holly Windows XP 32-bit holly talos other-e10s_nol64 ScriptFactory
-holly Ubuntu HW 12.04 x64 holly talos other-e10s_l64 ScriptFactory
-holly Ubuntu HW 12.04 x64 holly talos svgr-e10s ScriptFactory
-holly Ubuntu HW 12.04 x64 holly talos dromaeojs-e10s ScriptFactory
-holly Ubuntu HW 12.04 x64 holly talos chromez-e10s ScriptFactory
-holly Ubuntu HW 12.04 x64 holly talos tp5o-e10s ScriptFactory
-holly Ubuntu HW 12.04 x64 holly talos g1-e10s ScriptFactory
-holly Ubuntu HW 12.04 holly talos other-e10s_nol64 ScriptFactory
-holly Ubuntu HW 12.04 holly talos svgr-e10s ScriptFactory
-holly Ubuntu HW 12.04 holly talos dromaeojs-e10s ScriptFactory
-holly Ubuntu HW 12.04 holly talos chromez-e10s ScriptFactory
-holly Ubuntu HW 12.04 holly talos tp5o-e10s ScriptFactory
-holly Ubuntu HW 12.04 holly talos g1-e10s ScriptFactory

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Chris Peterson (:cpeterson) from comment #0)
> 2. After e10s is enabled by default in Nightly, will we test e10s and
> non-e10s on every push?

I can't see any reason to continue testing non-e10s after we enable it by default, unless we think there's a significant chance that we'll turn it off by default again.

Comment 7

[Chris Peterson [:cpeterson]]

(In reply to Ben Hearsum [:bhearsum] from comment #6)
> (In reply to Chris Peterson (:cpeterson) from comment #0)
> > 2. After e10s is enabled by default in Nightly, will we test e10s and
> > non-e10s on every push?
> 
> I can't see any reason to continue testing non-e10s after we enable it by
> default, unless we think there's a significant chance that we'll turn it off
> by default again.

e10s is currently enabled by default on Nightly, but it will not ride to Aurora. We need continue testing non-e10s on mozilla-central to avoid surprise regressions when mozilla-central is merged to Aurora where e10s is no longer enabled by default.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8526084 [details] [diff] [review]
pgo_talos-buildbot-configs.diff

Review of attachment 8526084 [details] [diff] [review]:
-----------------------------------------------------------------

catlee, thanks for writing this patch and answering my questions.  I look forward to this going live and us getting closer to e10s for everything.  Regarding e10s vs non e10s testing, users can disable this feature, I can see us testing non e10s on nightlies or in some limited fashion for a few release cycles after we have already released e10s on the release channel.

Comment 9

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/e3fc54de6e7e

Comment 10

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/60e31b9c1457

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

to wrap up this bug:
* turn off the winxp stuff - we are perma fail there
* determine if it is simple to get osx running (we don't do pgo for osx, so it didn't get scheduled)

some actions or reasons related to the above mentioned two items would be nice to see so we can close this bug out

Comment 12

[Chris Peterson [:cpeterson]]

(In reply to Joel Maher (:jmaher) from comment #11)
> to wrap up this bug:
> * turn off the winxp stuff - we are perma fail there
> * determine if it is simple to get osx running (we don't do pgo for osx, so
> it didn't get scheduled)

catlee: are these tasks on your plate? Do you need a developer from the e10s team to investigate the WinXP perma fail?

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

alright, winxp tests will be ready to go here soon- we run all suites on xp except for dromaeo and g1.

I am not sure how to get osx tests up and running on this schedule.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

ok, winxp fix is on m-c, likewise all the other tests should be converted (session restore is still broken though).

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

what are the next steps here?  Are we ready to run this more frequently?  I don't recall any regressions specific to e10s talos, so maybe there isn't a need to go forward.

My impression is that we need to get osx running e10s, once that is in place- we can have a better idea of what is working and what differences we have on our existing tests between e10s/non-e10s.

Comment 16

[Chris AtLee [:catlee]]

Is this still blocked by getting OSX running e10s?

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

we have osx running e10s tests.  We still need to get e10s scheduled on inbound/fx-team, even if we don't run it there.  Then at least we have the ability to backfill.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: add e10s talos jobs to inbound and fx-team (1.0)

ok, we are finding more and more e10s only regressions, this needs to get resolved one way or another to avoid wasting days of time pushing to try and waiting.

this adds all (but xperf) talos e10s jobs to inbound[+pgo] and fx-team[+pgo].  I know this isn't ideal for buildername limits and available machines, but without anybody adding a patch to solve this you get what you get!

ideally we could force coalescing with SETA or some other means, and then backfill as needed which would be 100% acceptable.

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: differences.txt when adding e10s talos builders to inbound/fx-team

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

Thanks catlee.  let me get this in and then we can determine ways to force coalescing if needed.

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/build/buildbot-configs/rev/ffd6aedf3243

Comment 22

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/ffd6aedf3243

Comment 23

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: disable osx g1 tests for talos e10s (1.0)

as these are permafail for months, lets disable them, ideally I will enable all of this on try server so folks can debug there.  For now lets free up some hours of osx 10.10 machine time.

here is the builder differences:
Builders removed
- Rev5 MacOSX Yosemite 10.10 fx-team talos g1-osx-e10s
- Rev5 MacOSX Yosemite 10.10 mozilla-central talos g1-osx-e10s
- Rev5 MacOSX Yosemite 10.10 mozilla-inbound talos g1-osx-e10s

Comment 24

[Justin Wood (:Callek)]

Comment on attachment 8644536 [details] [diff] [review]
disable osx g1 tests for talos e10s (1.0)

Review of attachment 8644536 [details] [diff] [review]:
-----------------------------------------------------------------

not a huge fan of commenting out in this file, since this file is already pretty crufty, but assuming its temporary and you're tracking it I'm ok with it

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/build/buildbot-configs/rev/574f0cc2646e

Comment 26

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/574f0cc2646e