1126493 - rollout 10.10 tests in a way that doesn't impact wait times

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

We aren't in the state yet where we can enable 10.10 tests on trunk (see bug 1121199 for status on greening up tests) but I thought I'd open this bug to get things started.

We don't currently have enough capacity to run tests on 10.10 and 10.8.  10.10 is the dominant platform compared to 10.8 for our users now.  Jake is going to find out when the ~25 used minis that Amy ordered are due to arrive and how much time dcops needs to get them up and running.  There is also the time needed to image them and enter their records into inventory etc.

So the question is:
Once we have 10.10 tests in a greener state, my thought is to disable the 10.8 
tests on trunk and enable the corresponding 10.10 tests so wait times in the 10.8 pool don't get worse. Also, if the new minis are delayed in arriving, we can still stand up the 10.10 tests. If they do arrive on time this plan still makes sense give the current long wait times for 10.8. Thoughts?

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1125998.patch

talked to coop about this a bit in our 1x1 today, he suggested ensuring we have green tests on aurora before enabling them on trunk branches etc.  In any case, he is a patch for when we get to that state

Comment 2

[Kim Moir [:kmoir] ET]

Attachment: bug1125998builder.diff

builder diff

Comment 3

[Amy Rich [:arr] [:arich]]

Approval at the Cxx level is still pending before we purchase the mac minis.

Comment 4

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-2.patch

patch to enable opt on 10.10 on trunk while keep 10.8 running on debug on these branches as suggested here

https://bugzilla.mozilla.org/show_bug.cgi?id=1131269#c4

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: bug1126493builder.diff

builder diff 

still have to figure out a way to disable talos on 10.8 and enabled on 10.10

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-reducemtnlion.patch

Was talking to coop today about how to reduce our load on mountainlion temporarily given that it looks like we don't have many new minis for 10.10.

One way is to reduce the frequency at which they run to we can pull some minis and reimage them for 10.10.  

This is an initial patch, if this doesn't change the load that much, we can reduce the frequency on other branches.

Comment 7

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/ea62c9e7828c

Comment 8

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-3.patch

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: bug1126493builder.diff

Comment 10

[Kim Moir [:kmoir] ET]

Comment on attachment 8565977 [details] [diff] [review]
bug1126493-3.patch

This patch is for enabling 10.10 opt tests on trunk once we have the machines reimaged in bug 1134223

Comment 11

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8565977 [details] [diff] [review]
bug1126493-3.patch

Review of attachment 8565977 [details] [diff] [review]:
-----------------------------------------------------------------

An overarching comment for this section indicating what our current plan is for 10.10 testing would not be amiss.

r+ with nits fixed.

::: mozilla-tests/config.py
@@ +2177,5 @@
>                  tests[3] = [x for x in tests[3] if x not in platforms_for_os or x in enabled_platforms_for_os]
>                  BRANCHES[branch]['%s_tests' % s] = tuple(tests)
>  
> +# bug 1126493 Enable Yosemite testing on select branches only
> +exclude_yosemite = ['try']

I think this var is poorly named. Based on the comment, it should be either include_yosemite or exclude_mountainlion.

@@ +2192,5 @@
> +if len(exclude_yosemite) > 0:
> +    delete_slave_platform(BRANCHES, PLATFORMS, {'macosx64': 'yosemite'}, branch_exclusions=exclude_yosemite)
> +
> +#opt and talos can be disabled on branches where they are enabled on 10.10
> +for branch in exclude_yosemite:

Could you nest this under the |if len(exclude_yosemite) > 0:| conditional above for clarity?

Comment 12

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-4.patch

fixed with feedback from review

Comment 13

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-5.patch

enable on 39 instead since merge happened.  Will run more tests to see current state of aurora

Comment 14

[Kim Moir [:kmoir] ET]

So I talked to the sheriffs today and they want a 50/50 split on 10.8/10.10 slave pools before this 10.10 is enabled on trunk.  So I opened bug 1137749 with relops to reimage some more machines. We will do this on Monday so we can get the patches for talos on graph servers enabled etc.

Also, I'm working on a patch to disable the 10.10 talos tests that erroneously get enabled on branches where don't have other 10.10 tests running.

Comment 15

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-6.patch

much better patch - ensures that talos tests that are enabled on 10.10 are removed on 10.8.  Also ensures talos tests are not enabled on m-a and m-b
will attach builder diff

Comment 16

[Kim Moir [:kmoir] ET]

Attachment: 10.10builder.diff

Comment 17

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8570671 [details] [diff] [review]
bug1126493-6.patch

Review of attachment 8570671 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +2192,5 @@
> +            if slave_platform not in ['mountainlion', 'yosemite']:
> +                continue
> +            if name not in include_yosemite:
> +                include_yosemite.append(name)
> +if len(include_yosemite) > 0:

You shouldn't need this condition if you are setting |include_yosemite = ['try']| above.

Comment 18

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-7.patch

patch with review comment incorporated

Comment 19

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-7.patch

Comment 20

[Kim Moir [:kmoir] ET]

Attachment: bug1126493debug.patch

patch to enable debug tests, work in bug 1125998 looks promising to reducing the time needed to run debug tests

Comment 21

[Kim Moir [:kmoir] ET]

Attachment: builder1126493debug.diff

builder diff for patch to enable debug tests on 10.10 and disable on 10.8

Comment 22

[Kim Moir [:kmoir] ET]

We were going to deploy this today but arr noticed an issue with slaveapi and reboots of yoseimite slaves. dividehex is investigating a fix. We are planning to enable 10.10 on trunk for opt tests tomorrow.  I'll disable the next batch of slaves starting for noon eastern or 9 am pacific.

Comment 23

[Kim Moir [:kmoir] ET]

Attachment: bug1126493-8.patch

unbitrotten patch

Comment 24

[Ryan VanderMeulen [:RyanVM]]

We're seeing 10.10 debug M(JP) and M(gl) jobs being scheduled on trunk. I've hidden M(JP) for now (and probably will hide gl too), but we should really kill those ASAP since we don't have the capacity to spare right now.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Job names are:
Rev5 MacOSX Yosemite 10.10 mozilla-central debug test mochitest-jetpack
Rev5 MacOSX Yosemite 10.10 mozilla-central debug test mochitest-gl

Presumably happening on other branches as well, but they haven't been enabled in production long enough to confirm that yet.

Comment 26

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/181c813fdbbb

Comment 27

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/09077cc77b32

Comment 28

[Kim Moir [:kmoir] ET]

Thanks RyanVM I'll write a patch to disable them.

Comment 29

[Kim Moir [:kmoir] ET]

Attachment: bug1126493disablegljp.patch

will attach builder diff

Comment 30

[Kim Moir [:kmoir] ET]

Attachment: bug1126493disablegljpbuilder.diff

patch to disable mochitest-gl and mochitest jetpack on debug for 10.10

Comment 31

[Kim Moir [:kmoir] ET]

Comment on attachment 8572312 [details]
bug1126493disablegljpbuilder.diff

previous comment should have read builder diff to disable mochitest-gl and mochitest jetpack on debug for 10.10

Comment 32

[Kim Moir [:kmoir] ET]

Attachment: bug1126493disablegljp-2.patch

There is an issue on the masters where 10.8 talos jobs are still being scheduled.  I did have this addressed on my dev-master but it seemed when I made a new patch this morning to account for bit rot, the order was lost and the part to remove mountain lion as a talos slave platform on trunk moved to after talos tests were loaded.  Anyways this patch addresses this + removes m-jp and m-gl on debug for 10.10

Comment 33

[Kim Moir [:kmoir] ET]

Attachment: bug1126493debugm-a.patch

patch to enable 10.10 tests on debug and m-a, only remaining 10.8 tests would be on branches where gecko < 37.  We are not ready to land this patch yet because of bug 1125998 (fix debug tests so the run in a reasonable timeframe) and bug 1139002 (green up tests on aurora)

Will attach a builder diff

Comment 34

[Kim Moir [:kmoir] ET]

Attachment: m-adebugbuilder.diff

Comment 35

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/c538e70cbcf9

Comment 36

[Kim Moir [:kmoir] ET]

Once bug 1137963 is resolved (and a few tests are fixed) I should be able to enable 10.10 tests on aurora and debug on trunk.  At the same time I'll disable the corresponding 10.8 tests.  At that point, I'd like to reimage another 25 10.8 machines as 10.10.  This will allow leave us with 25 10.8 machines for beta and release tests. 

Once 10.10 tests are on beta, I'd like to reimage 15 more machines so there are only 10 10.8 test machines left for release builds.  :rail, does this seem reasonable to you?  I thought I'd ask since you are on release duty.

Comment 37

[Ryan VanderMeulen [:RyanVM]]

We talked about this on IRC a bit. Now that all 10.8 jobs have been shut off on the B2G release branches, that'll leave us just needing to support 10.8 on mozilla-release and mozilla-esr31 once Gecko 38 hits beta. We run about 30 10.8 test jobs per push. In general, Windows PGO builds/tests are the long pole for overall run to completion time on the release branches, and neither of those branches generally see significant push volume on a daily basis.

Assuming we have the ability to reimage from 10.10 back to 10.8 if the wait times are too long, could we even try leaving only 5 10.8 slaves as a first shot? We'll still have 10.6 tests for faster turnaround too. If that proves too few, maybe go up to 10. But overall, I think we should err on the side of 10.10 capacity as much as we can get away with.

Comment 38

[Kim Moir [:kmoir] ET]

removing rail needinfo since RyanVM responded to question

Comment 39

[Kim Moir [:kmoir] ET]

Attachment: bug1126493debugonly.patch

patch to enable on debug only since patch to fix debug hasn't landed on aurora yet

Comment 40

[Kim Moir [:kmoir] ET]

Attachment: bug1126493debugonly-2.patch

change patch so all 10.8 tests remains an option on try

Comment 41

[Kim Moir [:kmoir] ET]

Attachment: 10.10builder.diff

builder diff for latest patch to enable debug on trunk + keep 10.8 running on try

Comment 42

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/b709d2e3a153

Comment 43

[Kim Moir [:kmoir] ET]

10.10 debug are now running on trunk and there are now 25 additional 10.10 machines we enabled in bug 1140246

Comment 44

[Kim Moir [:kmoir] ET]

Attachment: bug1126493aurora.patch

We can enable 10.10 on aurorar now that the patch in bug 1138616 has been uplifted

Comment 45

[Kim Moir [:kmoir] ET]

Attachment: bug1126493aurora.diff

builder diff from previous patch

Comment 46

[Kim Moir [:kmoir] ET]

From irc
philor	kmoir: and speaking of disused filing cabinets in the basement, https://treeherder.mozilla.org/#/jobs?repo=addon-sdk is still using 10.8 when it should have switched to 10.10 along with the trunk
kmoir	philor: okay didn't know about that, will look at that
philor	kmoir: it hides in two places, weird conditionals in http://mxr.mozilla.org/build/source/tools/buildfarm/utils/run_jetpack.py?force=1 and... somewhere in configs, can't remember where

I have patches to fix, will attach

Comment 47

[Kim Moir [:kmoir] ET]

Attachment: bug1126493jp.patch

switch jetpack to use yosemite instead of mtnlion
builder diff

(test16)[[e-mail] test16]$  diff old2 new2
3479,3480c3479,3480
< jetpack-fx-team-mountainlion-opt ScriptFactory
< jetpack-fx-team-mountainlion-debug ScriptFactory
---
> jetpack-fx-team-yosemite-opt ScriptFactory
> jetpack-fx-team-yosemite-debug ScriptFactory

Comment 48

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/f4253de25db9

Comment 49

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/c667541a6585

Comment 50

[Kim Moir [:kmoir] ET]

closing this bug, thanks everyone for your help

On next merge day, we'll have to enable talos yosemite tests on beta, the patch is in bug 1144102